Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 448.974537ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:15:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 449.005788ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................XX...X........] duration: 464.166853ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:15:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 464.263441ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.057737419s trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:15:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.057962821s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [......X..XXXXXXXXXXXXXXXXXXXX...............................] duration: 2.042162485s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:15:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.042201565s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XX...X.......] duration: 469.63765ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:15:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 469.665125ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 339.667633ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:14:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 339.686214ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 462.340819ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:14:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 462.367622ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 421.594694ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:14:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 421.635937ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................XX...X......] duration: 465.74584ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:14:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 465.774955ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 436.035773ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:14:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 436.07629ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [.......X..XXXXXXXXXXXXXXXXXXXX..............................] duration: 2.101983299s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:14:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.102013139s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 440.57458ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:13:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 440.65315ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XX...X.....] duration: 430.055513ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:13:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 430.088013ms edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [........X..XXXXXXXXXXXXXXXXXXXX.............................] duration: 2.509353697s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:13:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.509390977s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 449.506284ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:13:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 449.547103ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 497.739166ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:13:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 497.774201ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 497.796502ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:13:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 497.887259ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................XX...X....] duration: 547.73183ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:12:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 547.757869ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 490.056833ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:12:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 490.076081ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [.........X..XXXXXXXXXXXXXXXXXXXX............................] duration: 2.520463766s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:12:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.520493991s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................XX...X...] duration: 457.966088ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:12:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 458.094684ms edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [..........X..XXXXXXXXXXXXXXXXXXXX...........................] duration: 2.485416197s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:11:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.485552986s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 510.550151ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:11:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 510.587614ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 453.228857ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:11:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.256761ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 462.980393ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:11:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 463.009317ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [...........X..XXXXXXXXXXXXXXXXXXXX..........................] duration: 2.465073509s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:11:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.465179004s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XX...X..] duration: 532.05701ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:11:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 532.091013ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 528.197524ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:10:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 528.219017ms core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [............X..XXXXXXXXXXXXXXXXXXXX.........................] duration: 2.497203657s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:10:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.497319075s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XX...X.] duration: 931.797379ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:10:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 931.825112ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XX...X] duration: 30.005263746s trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:10:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.005292127s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.370246061s trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:10:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.370282462s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 45.136492168s trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:10:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 45.136603191s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.027532547s trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:10:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.027557292s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [.............X..XXXXXXXXXXXXXXXXXXXX........................] duration: 2.529137367s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:10:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.529171347s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 525.520156ms trace: https://jaeger.dev.storj.io/trace/777cf703997eb9bf FAIL (2026-01-28 13:09:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 525.619364ms edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [..............X..XXXXXXXXXXXXXXXXXXXX.......................] duration: 2.137499472s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:08:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.137529934s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 479.787262ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:08:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 479.829178ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 471.925622ms trace: https://jaeger.dev.storj.io/trace/777cf703997eb9bf FAIL (2026-01-28 13:07:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 471.950163ms edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [...............X..XXXXXXXXXXXXXXXXXXXX......................] duration: 2.071642442s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:07:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.071673004s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 396.066367ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:07:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 396.098561ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 464.938782ms trace: https://jaeger.dev.storj.io/trace/777cf703997eb9bf FAIL (2026-01-28 13:07:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 464.972858ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.290935451s trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:07:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.290971625s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.035299794s trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:07:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.035329699s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [................X..XXXXXXXXXXXXXXXXXXXX.....................] duration: 2.549853797s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:07:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.549882837s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.000618971s trace: https://jaeger.dev.storj.io/trace/777cf703997eb9bf FAIL (2026-01-28 13:06:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000641358s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [.................X..XXXXXXXXXXXXXXXXXXXX....................] duration: 2.160040847s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:06:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.160074598s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001069532s trace: https://jaeger.dev.storj.io/trace/2d3f405876bfc122 FAIL (2026-01-28 13:06:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001187867s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [..................X..XXXXXXXXXXXXXXXXXXXX...................] duration: 2.540476443s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:05:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.540504168s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 21 out of 60 is failed (threshold=3) [...................X..XXXXXXXXXXXXXXXXXXXX..................] duration: 2.08273016s trace: https://jaeger.dev.storj.io/trace/4537c4dcc51765d5 FAIL (2026-01-28 13:04:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.235438033 (308h56m2.718895241s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237854277 (308h56m2.721311474s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.237961780 (308h56m2.721418983s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.239706676 (308h56m2.723163882s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.240195941 (308h56m2.723653148s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.241552011 (308h56m2.725009212s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.242344396 (308h56m2.725801605s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.243249502 (308h56m2.726706703s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244507175 (308h56m2.727964383s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.244848450 (308h56m2.728305647s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.246412591 (308h56m2.729869795s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Uploading piece... Jan 28 12:51:09.305554531 (308h56m2.789011739s) [noise=true , address=185.167.204.205:4020 , piece_id=H7FROURBL5GP5WO66A64JSXIZ4ONSWDSQ7XS6FEYCVN67BLOF7TQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12hUxMR4vpUXuoAToGJcWvzaXsU4BcV5BFicueEaYBgyVLPQDoU, address:185.167.204.205:4020): protocol: write tcp 159.223.143.39:38130->185.167.204.205:4020: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.317750782 (308h56m2.80120798s) [noise=true , address=185.167.204.205:4002 , piece_id=RCONJMLZFG23TTEBUVW7QC7L4DEAZQP2G5OWET3P2TIS2WFIUV2A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: write tcp 159.223.143.39:51148->185.167.204.205:4002: write: connection refused; conn closed: op Jan 28 12:51:09.327347049 (308h56m2.810804249s) [noise=true , address=185.167.204.205:4005 , piece_id=LZBLWNMJNALF5R3VBANONRO5HC6TXDQYGQD7H7FCJ3RULMXZWUJA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12B18LegNdvxjRqBgwnFB4Aig1nt5URm9Au7bWXsfbBPyCnterM, address:185.167.204.205:4005): protocol: write tcp 159.223.143.39:50500->185.167.204.205:4005: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 12:51:09.653950714 (308h56m3.137407902s) [upload=321189]: Next returned nil segment Jan 28 12:51:09.653964663 (308h56m3.137421861s) [upload=321189]: Waiting for error group managing segments... Jan 28 12:51:09.837396638 (308h56m3.320853835s) [noise=true , address=185.167.204.206:4065 , piece_id=6U5J64GUQSUXXPNRDPZ7EL5KPR25HWYFTZGCFSSZQ5EEFVQL3INQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.839751104 (308h56m3.323208305s) [noise=true , address=89.149.201.147:4004 , piece_id=FSBBD5YUVGC5DBKQSTNRBMH7ONWK4SX7Q37IBIMKP5PGLHLNXTQQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.848074501 (308h56m3.331531699s) [noise=true , address=81.171.14.70:4046 , piece_id=SXPK4YXDOLR64VDRNVMO2QR6BC67NKVA7GWAHD6V2KLSGJFU76CA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.849109167 (308h56m3.33256636s) [noise=true , address=89.149.201.147:4010 , piece_id=XESXIGOWR4NIEOWI5Z5E4Q2QJB5PNVBH7ZP5F2LXR74UJ3MDYFRA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.854916364 (308h56m3.338373552s) [noise=true , address=81.171.14.70:4048 , piece_id=6HPAQZQYKOZJU4CPSD5KD2PVZZJRQLJVPNVPOSPR7P5OU4E4QOCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.857014024 (308h56m3.340471208s) [noise=true , address=185.167.204.206:4059 , piece_id=HNYYWIKIUPYUYPC3H5ZWF6VPHMRV7CP54XY3XH4CBTIU32EIJTJQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.864986055 (308h56m3.348443246s) [noise=true , address=185.167.204.205:4022 , piece_id=RAYH77FBW4YHBL74IR44H5YLOCSBAMBMRPRPBC3VI3LEOW3CTGSQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.879225162 (308h56m3.362682353s) [noise=true , address=81.171.14.70:4028 , piece_id=BPJDDF4CYJYTA4N3XZXUOTTTX3JNJ3SMQLX5WX7C5Y56K74NIMHA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.887010999 (308h56m3.370468201s) [noise=true , address=81.171.14.70:4032 , piece_id=R2OIIUHW24J5RHEQZKDJQGVSRUW2OEEFPQMCM6JQTS424JQDMGIQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.890495852 (308h56m3.373953049s) [noise=true , address=89.149.201.147:4024 , piece_id=4UF67MQDTWEV7AP4HFXVCF7KBAYMKHEM3DXSSOL67E6J6SU47AGA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891424018 (308h56m3.374881215s) [noise=true , address=185.167.204.206:4068 , piece_id=WPX6CTEA7OTQ22PJWVX7HQ2UAAKEUAI7W5QEYETSZM7OS3OTV5VQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.891513529 (308h56m3.374970717s) [noise=true , address=89.149.201.147:4022 , piece_id=VK56BEILSN3V3OKM7SWOECLA2LYAFBGYJAOKRPDBZTFV6QWRH5ZQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.893166674 (308h56m3.376623872s) [noise=true , address=81.171.14.70:4040 , piece_id=UPVQO357MRJ6DOZEIFB526VHJ3MLR5OYLRN6OHAQW7HRR5IUPFBQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.901033954 (308h56m3.384491152s) [noise=true , address=185.167.204.205:4034 , piece_id=QDJPOPAYWN3MB26Z6PJZCKIJXN6IVKMVTFZGRSE4G6SC7DJQXNCQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.902384800 (308h56m3.385841998s) [noise=true , address=185.167.204.206:4055 , piece_id=EKRIDDHG2NQTTISBNDGVHH5UQTEKYBUTR52MSHW54HSSFHJCQFWQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.903445897 (308h56m3.386903095s) [noise=true , address=81.171.14.70:4047 , piece_id=TP5ZHOVLRNGJSQV2Z2UGEVUXBB3S2EQFT7OGG3DZ3GKGSUQNPPKQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904192288 (308h56m3.387649472s) [noise=true , address=81.171.14.70:4039 , piece_id=KPWTL2OONJVGEPHZBCHVBQLB6TYDIZXKUS2AU2ULFL6FARPGV52A, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.904389755 (308h56m3.387846953s) [noise=true , address=185.167.204.205:4003 , piece_id=EKAQFUFATTAAOB3ED2Y7T363GF7QSIHOQBHSAEW5XUJOXHDNAQUQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.906709761 (308h56m3.390166945s) [noise=true , address=89.149.201.147:4012 , piece_id=3IGZ3PFTJVJIGY5DJ4KBYSYGWTD34GJZSJWJM4J4PDGV3EQVTDSA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916653501 (308h56m3.4001107s) [noise=true , address=185.167.204.206:4057 , piece_id=BKYRXOXRCUDQQA4USEFI7CSI3AIQDDFEVFVKMWJRALEWQ4YPF2EA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.916829692 (308h56m3.400286894s) [noise=true , address=185.167.204.206:4052 , piece_id=4DCC4BJU7EBSFAOQ23ILQ6LZIWB6M2ZSFLUHNLCY2IQ6CZOBOQVQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:09.947096963 (308h56m3.430554154s) [noise=true , address=81.171.14.70:4035 , piece_id=2IDPBOBZ7DPBKJMFCDBRY3UL2HJAL3IJT5PDKCU47S4DD7SDOCNQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.313930045 (308h56m3.797387247s) [noise=true , address=89.149.201.147:4021 , piece_id=MJWNWMEJSBOSEUNKJDNJD6N73AMVMSNSZODXSQ2F2OMQ6YU55XMA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:51:10.355312826 (308h56m3.838770026s) [noise=true , address=81.171.14.70:4033 , piece_id=K52QMG6336HEWAIOPCTY3FHMCO46MESMYPJDDOBFUYCGBLNOMZWA, seg_pos={0 0}, upload=321189]: Done uploading piece. err: <nil> Jan 28 12:52:08.840209698 (308h57m2.323666907s) [noise=true , address=185.167.204.205:4019 , piece_id=UD4KRYOVXOENA572WJH2GVGNE2TQQC5MQPG3ZVVCHWVZTEL4H6NQ, seg_pos={0 0}, upload=321189]: Done uploading piece. err: ecclient: upload failed (node:12CwWyCCjbNNuuSxpk9PhiWxG2yvuT19gDyJhezMfUEez6LrpDj, address:185.167.204.205:4019): protocol: context deadline exceeded; context deadline exceeded Jan 28 12:52:08.840255268 (308h57m2.323712455s) [seg_pos={0 0}, upload=321189]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Jan 28 12:52:08.937080947 (308h57m2.420538134s) [upload=321189]: Done uploading segments... Jan 28 12:52:08.937118447 (308h57m2.42057565s) [upload=321189]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.082930054s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)