Orbiter History

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 365.690994ms trace: https://jaeger.dev.storj.io/trace/799201ad3eed1a9e FAIL (2025-12-11 02:15:05 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 365.743436ms edge authservice http bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 472.299598ms trace: https://jaeger.dev.storj.io/trace/e00af73289d0d52 FAIL (2025-12-11 02:14:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 403, RequestID: 1880072F15065F7F, HostID: , api error InvalidAccessKeyId: The Access Key Id you provided does not exist in our records. Duration: 472.333379ms core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 365.218471ms trace: https://jaeger.dev.storj.io/trace/799201ad3eed1a9e FAIL (2025-12-11 02:14:28 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 365.240424ms edge authservice http bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 560.604804ms trace: https://jaeger.dev.storj.io/trace/e00af73289d0d52 FAIL (2025-12-11 02:14:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 403, RequestID: 1880072F15065F7F, HostID: , api error InvalidAccessKeyId: The Access Key Id you provided does not exist in our records. Duration: 560.632667ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 468.055747ms trace: https://jaeger.dev.storj.io/trace/e00af73289d0d52 FAIL (2025-12-11 02:13:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 403, RequestID: 1880072F15065F7F, HostID: , api error InvalidAccessKeyId: The Access Key Id you provided does not exist in our records. Duration: 468.081709ms core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 379.492263ms trace: https://jaeger.dev.storj.io/trace/799201ad3eed1a9e FAIL (2025-12-11 02:13:30 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 379.517457ms edge authservice http bgp eu1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 568.484095ms trace: https://jaeger.dev.storj.io/trace/799201ad3eed1a9e FAIL (2025-12-11 02:13:06 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 568.520393ms edge authservice http bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 401.211658ms trace: https://jaeger.dev.storj.io/trace/e00af73289d0d52 FAIL (2025-12-11 02:13:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 403, RequestID: 1880072F15065F7F, HostID: , api error InvalidAccessKeyId: The Access Key Id you provided does not exist in our records. Duration: 401.242471ms core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 391.440053ms trace: https://jaeger.dev.storj.io/trace/799201ad3eed1a9e FAIL (2025-12-11 02:12:37 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 391.469551ms edge authservice http bgp eu1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 370.127376ms trace: https://jaeger.dev.storj.io/trace/799201ad3eed1a9e FAIL (2025-12-11 02:12:10 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 370.152548ms edge authservice http bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 592.493329ms trace: https://jaeger.dev.storj.io/trace/e00af73289d0d52 FAIL (2025-12-11 02:12:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 403, RequestID: 1880072F15065F7F, HostID: , api error InvalidAccessKeyId: The Access Key Id you provided does not exist in our records. Duration: 592.523106ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 484.879634ms trace: https://jaeger.dev.storj.io/trace/e00af73289d0d52 FAIL (2025-12-11 02:11:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 403, RequestID: 1880072F15065F7F, HostID: , api error InvalidAccessKeyId: The Access Key Id you provided does not exist in our records. Duration: 484.901781ms core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 391.389777ms trace: https://jaeger.dev.storj.io/trace/799201ad3eed1a9e FAIL (2025-12-11 02:11:06 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 391.425532ms edge authservice http bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 425.108936ms trace: https://jaeger.dev.storj.io/trace/e00af73289d0d52 FAIL (2025-12-11 02:10:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 403, RequestID: 1880072F15065F7F, HostID: , api error InvalidAccessKeyId: The Access Key Id you provided does not exist in our records. Duration: 425.133295ms core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 377.116948ms trace: https://jaeger.dev.storj.io/trace/799201ad3eed1a9e FAIL (2025-12-11 02:10:38 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 377.151643ms edge authservice http bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 465.410201ms trace: https://jaeger.dev.storj.io/trace/e00af73289d0d52 FAIL (2025-12-11 02:10:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 403, RequestID: 1880072F15065F7F, HostID: , api error InvalidAccessKeyId: The Access Key Id you provided does not exist in our records. Duration: 465.427658ms core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 391.582089ms trace: https://jaeger.dev.storj.io/trace/799201ad3eed1a9e FAIL (2025-12-11 02:10:06 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 391.608787ms edge authservice http bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 142.809677ms trace: https://jaeger.dev.storj.io/trace/e00af73289d0d52 FAIL (2025-12-11 02:09:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 403, RequestID: 1880072F15065F7F, HostID: , api error InvalidAccessKeyId: The Access Key Id you provided does not exist in our records. Duration: 142.832466ms core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 375.876423ms trace: https://jaeger.dev.storj.io/trace/799201ad3eed1a9e FAIL (2025-12-11 02:08:57 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 375.900198ms edge authservice http bgp eu1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 17.844889ms trace: https://jaeger.dev.storj.io/trace/799201ad3eed1a9e FAIL (2025-12-11 02:08:01 UTC) orbiter warning: unexpected status code "400 Bad Request" "Client sent an HTTP request to an HTTPS server.\n" Duration: 17.873935ms edge authservice http bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 469.743968ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:33:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 469.77228ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 430.387396ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:33:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 430.427285ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 403.926608ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:32:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 403.980653ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 501.366626ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:32:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 501.398078ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 405.851671ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:31:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 405.879367ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 423.005842ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:31:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 423.040601ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 611.072222ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:31:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 611.098954ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration: 473.341839ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:30:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 473.35941ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXXXXXX.....................................................] duration: 649.176702ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:30:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 649.19866ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 1.021947038s trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:30:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1.021975768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXXXXXXX....................................................] duration: 381.169179ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:29:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 381.195954ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 628.27679ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:28:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 628.296348ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XXXXXXXX...................................................] duration: 1.497915475s trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:28:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.497950268s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.XXXXXXXX..................................................] duration: 400.422835ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:28:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 400.47413ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 679.442917ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:27:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 679.490112ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration: 566.381775ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:27:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 566.409739ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX.XXXXXXXX.................................................] duration: 1.853979371s trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:26:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.854008182s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXX.XXXXXXXX................................................] duration: 471.566907ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:26:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 471.594854ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXXXXXX.....................................................] duration: 536.870192ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:26:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 536.896286ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXX.XXXXXXXX...............................................] duration: 777.989519ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:25:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 778.018005ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XXXXXXX....................................................] duration: 956.216258ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:25:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 956.247578ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXX..............................................] duration: 494.829781ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:24:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 494.85808ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.XXXXXXX...................................................] duration: 531.615445ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:24:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 531.641195ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXX.............................................] duration: 1.003858888s trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:24:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.003895298s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XXXXXX.XXXXXXXX............................................] duration: 413.84251ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:23:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 413.868216ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX.XXXXXXX..................................................] duration: 509.157354ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:23:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 509.254852ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 438.905832ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:23:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 438.929424ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXX.XXXXXXX.................................................] duration: 473.460585ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:23:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 473.556804ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXX.XXXXXXX................................................] duration: 482.319524ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:22:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 482.350228ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.XXXXXX.XXXXXXXX...........................................] duration: 397.968936ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:22:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 398.001482ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777"} failure (antiblip window 3/60): (authservice.eu1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)