Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 6 out of 60 is failed (threshold=3) [..............................X......X......XX.X...........X] duration: 1m0.100077576s trace: https://jaeger.dev.storj.io/trace/3b4460442365c0e7 FAIL (2026-03-10 19:28:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389309232 (104h27m50.986080258s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.389816473 (104h27m50.986587499s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.391931427 (104h27m50.988702464s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.394396629 (104h27m50.991167767s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.397263419 (104h27m50.994034452s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.399366031 (104h27m50.996137061s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.406080001 (104h27m51.002851053s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.409602698 (104h27m51.006373731s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.416502293 (104h27m51.013273329s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.420753887 (104h27m51.017524921s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.423262462 (104h27m51.020033496s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Uploading piece... Mar 10 19:27:53.823101372 (104h27m51.419872405s) [upload=116916]: Next returned nil segment Mar 10 19:27:53.823120446 (104h27m51.419891481s) [upload=116916]: Waiting for error group managing segments... Mar 10 19:27:54.000664652 (104h27m51.597435708s) [noise=true , address=81.171.14.70:4035 , piece_id=2YPHK6A2ZI6JHENYBHQXAQCHXVFSKC5E3RQC7PMWZKCJGB4RB64A, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.008012848 (104h27m51.604783892s) [noise=true , address=81.171.14.70:4029 , piece_id=4LYITJ27FYZX25WUL42RFJUX7VSDBA7CXJJASFKRAFGPXJYKM6QA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.011975049 (104h27m51.608746079s) [noise=true , address=81.171.14.70:4038 , piece_id=2HBQKDA5TGBX2P7NASC6NA4MWD7NVDUUG3VBAECSTG5TRDDR2I3Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013324482 (104h27m51.610095512s) [noise=true , address=185.167.204.205:4032 , piece_id=VXAHZM6L7462VFYIJFBL6RHMJAYOGY3EODRUOHWII4HKM255UGZA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.013614745 (104h27m51.610385775s) [noise=true , address=81.171.14.70:4045 , piece_id=MKI5ZLCAPRJRWQIWQLNP5RR6SQJXZZZAM35WDE6WVWQG2TPBMKXQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.026623408 (104h27m51.623394435s) [noise=true , address=81.171.14.70:4033 , piece_id=LWM6DL6LLEPLGWK3Q3QIYLRYEJA4FWT7OUN5WPSRXPERNKFZKIIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.035560337 (104h27m51.63233135s) [noise=true , address=89.149.201.147:4014 , piece_id=IH6DEVCPE7J7PQBSWJWIIDQWGS7PARMLMCOK3S25AXMSJFUVTUIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039223932 (104h27m51.635994945s) [noise=true , address=185.167.204.206:4052 , piece_id=AXXKU7XCFVOYI4KBLQ4P5JOCNVWTR27HDBB6LLCFMRFEKRE3VOQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.039642416 (104h27m51.636413439s) [noise=true , address=81.171.14.70:4025 , piece_id=JFDMQDIU7DOBV74FP27G7EYM4B5YDLNCP4ZMBEPLS2O37GKXH3VA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.041029145 (104h27m51.637800171s) [noise=true , address=81.171.14.70:4047 , piece_id=DRMYHFSFKE6W754ZVGR4443E4IN3RSLIDF4YVS4JLRWTHJJSGNRA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.044680567 (104h27m51.64145159s) [noise=true , address=81.171.14.70:4037 , piece_id=U5Z5435XRWAWRIAS5HFPZI3YHGPRVWPRQB2BSOCLZDO3NGTMKO5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.050626247 (104h27m51.647397275s) [noise=true , address=185.167.204.206:4058 , piece_id=NEUNBF5MTH7ZNJ3OKD2SCMOSILZ7THBMO6EYNHZ4IQNIQ5T3IJ7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.063548076 (104h27m51.6603191s) [noise=true , address=185.167.204.205:4014 , piece_id=7JROEIOUUMNHTLLV3S4D3Q2RJ6JEZZBF6GHJI3BE635Z7NRRPPYA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.080832704 (104h27m51.677603718s) [noise=true , address=185.167.204.206:4060 , piece_id=WX5QVUZU7F67SGJPGTEYVEKBHCYXWXI2JEOMLO5EQULFLFS4HRRQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.088180011 (104h27m51.684951025s) [noise=true , address=185.167.204.206:4059 , piece_id=CNTY6AKBW2GJCDKENAHK7MBZHGBAR6LIMBQH5LL7RWK3U7VC5XSQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.109755006 (104h27m51.706526034s) [noise=true , address=185.167.204.205:4035 , piece_id=PPE2DXQIF574R55F7BGW5SRV2MTAROPD2QERBVJ6IXKV4HMCIHFA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.117984644 (104h27m51.71475567s) [noise=true , address=185.167.204.205:4019 , piece_id=FXPKM5PNNZC5SEE3PBR7VPGX4Y644QCC6IBRJTF4PVRVUEC7XEQQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.118587921 (104h27m51.715358933s) [noise=true , address=185.167.204.206:4040 , piece_id=WBPAL2X62XGHRZ6KPNSHAXNE4U5XDXYTESCTVBW6F4G4JAF5JALQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.127117193 (104h27m51.723888222s) [noise=true , address=185.167.204.205:4011 , piece_id=ZX3ZEUKAWSONJTU26LMU6KJRFWFOLWHW4TPD3FIQHQLDZQFM34UA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.166283722 (104h27m51.763054748s) [noise=true , address=89.149.201.147:4003 , piece_id=LN2QSHLVIU3KNMBD7Z7D7EZYRWO7IESM7FJ5UAQGFX5ON5FP5J7Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.472291315 (104h27m52.069062329s) [noise=true , address=81.171.14.70:4040 , piece_id=GIBXDZKK4H5WKV4AR2VAA6BWYNPN3AYOD66DN4WGWZPXKXHRRW5Q, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.486254788 (104h27m52.083025819s) [noise=true , address=89.149.201.147:4013 , piece_id=X3IEGLOWJDQTXWRFFTEJGZFBD5TQJNURDIHP2RUGJE65YJUQAISQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:54.552368074 (104h27m52.149139108s) [noise=true , address=89.149.201.147:4023 , piece_id=JWOFVPWTRQ45HXXXKDRH2MDBKQL7MC6VNMBSLPBVJNDHDBE5XTAA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:56.435632635 (104h27m54.032403649s) [noise=true , address=185.167.204.206:4056 , piece_id=ROA2FCZ6WNQHF4IDG7LPZ2LG7BRBGX33DBUXBU6EOFYJYKCZCUTQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:27:58.800969733 (104h27m56.397740758s) [noise=true , address=185.167.204.205:4033 , piece_id=O2SIBP4ZSONAV6MUS6IDXXNM7PDLJQBQBELUOUBXZOSKLDSIR7NA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: <nil> Mar 10 19:28:53.009605827 (104h28m50.606376886s) [noise=true , address=89.149.201.147:4015 , piece_id=WR2GVYGD7PZISA7LP4PAEF4RFA2YZPF3VYETYDO6TZDOXSRQ2FXA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: read tcp 159.223.143.39:34090->89.149.201.147:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009693883 (104h28m50.606464943s) [noise=true , address=89.149.201.147:4017 , piece_id=J7PZSRWRGAF6Y5JZGF6G62STRQQSKC245UW4MOAEMGPEUMUU6SDA, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:33010->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009755582 (104h28m50.606526616s) [noise=true , address=89.149.201.147:4005 , piece_id=FUCA7DUJAK7UVOI7O4LJSYKLCOPRUAJW64DQH4JCVEZBLQMO4IIQ, seg_pos={0 0}, upload=116916]: Done uploading piece. err: ecclient: upload failed (node:14cSz6XLmfQYFmiCDqybBjNJGY9tUaNxsk8MfKc1ZHxZtRirZK, address:89.149.201.147:4005): protocol: read tcp 159.223.143.39:48940->89.149.201.147:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:28:53.009829068 (104h28m50.606600111s) [seg_pos={0 0}, upload=116916]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:28:53.106470711 (104h28m50.703241735s) [upload=116916]: Done uploading segments... Mar 10 19:28:53.106515494 (104h28m50.703286524s) [upload=116916]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 1m0.100103051s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...X...X...................X......X........................] duration: 1.089333441s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:28:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.089368706s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X....X..........X................................] duration: 1.297920894s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:28:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.297970142s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X....X..........X...............................] duration: 1.167480843s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:28:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.167508671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X...X...X...................X......X.......................] duration: 1.212047323s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:28:03 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.212076449s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 613.779562ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:27:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 613.811776ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X....X..........X..............................] duration: 1.347438113s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:27:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.34747784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...X...X...................X......X......................] duration: 1.218900486s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:27:34 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.218927009s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.............X....X..........X.............................] duration: 1.273706035s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:27:24 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.27373006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...X...X...................X......X.....................] duration: 1.097981522s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:27:10 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.0980123s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [...............................X......X......XX.X...........] duration: 22.049988437s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:27:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 22.050020481s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 783.30662ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:26:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 783.339821ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............X....X..........X............................] duration: 1.110697553s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:26:45 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.110726772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...X...X...................X......X....................] duration: 1.211047491s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:26:15 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.211075208s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 929.022683ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:26:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 929.050261ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.............X....X..........X...........................] duration: 1.342270095s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:25:51 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.342302989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [................................X......X......XX.X..........] duration: 2.761464794s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:25:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.76149451s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 616.406414ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:25:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 616.440967ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.............X....X..........X..........................] duration: 993.008709ms trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:25:22 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 993.110656ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.............X....X..........X.........................] duration: 1.313316962s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:25:06 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.313354175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X...X...X...................X......X...................] duration: 1.243515257s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:25:01 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.243552238s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [.................................X......X......XX.X.........] duration: 2.625859979s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:24:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.625886481s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.............X....X..........X........................] duration: 1.360628832s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:24:47 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.360656295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 721.349767ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:24:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 721.378427ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X...X...X...................X......X..................] duration: 1.281351163s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:24:21 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.281397751s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 635.311949ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:24:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 635.342147ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [..................................X......X......XX.X........] duration: 15.051968514s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:23:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 15.0519945s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.............X....X..........X.......................] duration: 1.290114981s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:23:54 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.290145472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.............X....X..........X......................] duration: 1.245307377s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:23:27 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.24533344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...X...X...................X......X.................] duration: 1.100524261s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:23:21 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.100555689s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 676.468222ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:23:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 676.492182ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [...................................X......X......XX.X.......] duration: 24.108231658s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:23:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 24.108264258s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.......X.............X....X..........X.....................] duration: 1.293411049s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:22:52 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.293441684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...X...X...................X......X................] duration: 1.245577327s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:22:41 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.245608686s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 715.314143ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:22:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 715.341972ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.......X.............X....X..........X....................] duration: 1.298326694s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:22:35 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.298354816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X...X...X...................X......X...............] duration: 1.241827998s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:22:23 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.241915031s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [....................................X......X......XX.X......] duration: 6.035940822s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:22:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 6.03597315s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X...X...X...................X......X..............] duration: 1.201433238s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:22:05 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.201453315s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 654.130938ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:21:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 654.158114ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.......X.............X....X..........X...................] duration: 1.268912203s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:21:47 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.268946038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [.....................................X......X......XX.X.....] duration: 11.760845058s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:21:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 11.760878045s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.......X.............X....X..........X..................] duration: 1.240051475s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:21:28 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.240074747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X...X...X...................X......X.............] duration: 1.407628233s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:21:12 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.407654854s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 721.817415ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:20:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 721.847078ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.......X.............X....X..........X.................] duration: 1.557299893s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:20:32 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.557330052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [......................................X......X......XX.X....] duration: 5.124048535s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:20:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.124078975s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.......X.............X....X..........X................] duration: 1.457986098s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:20:13 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.458038468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X...X...X...................X......X............] duration: 1.394902161s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:20:13 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.394920465s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 941.451664ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:20:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 941.550999ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)