Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 392.500411ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:48:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 392.551268ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 459.7289ms trace: https://jaeger.dev.storj.io/trace/78b5234b804c56b2 FAIL (2026-01-28 13:47:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (309h20m2.954159537s) [seg_pos={0 0}, upload=321587]: Begin upload segment... Jan 28 13:15:09.470766101 (309h20m2.954223302s) [seg_pos={0 0}, upload=321587]: Done begin upload segment. Jan 28 13:15:09.470798813 (309h20m2.954256012s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471157200 (309h20m2.954614403s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471670574 (309h20m2.955127779s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472180500 (309h20m2.955637705s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472712598 (309h20m2.956169803s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473275419 (309h20m2.956732619s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473668462 (309h20m2.957125659s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474056101 (309h20m2.957513306s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474452312 (309h20m2.957909509s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474782997 (309h20m2.958240194s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.480853378 (309h20m2.964310584s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.481720836 (309h20m2.965178042s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.483099633 (309h20m2.96655683s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.486563577 (309h20m2.970020774s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.492900551 (309h20m2.976357758s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.494672616 (309h20m2.978129817s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.496349386 (309h20m2.979806586s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.498142179 (309h20m2.981599388s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.570661945 (309h20m3.054119149s) [upload=321587]: Next returned nil segment Jan 28 13:15:09.570679413 (309h20m3.05413662s) [upload=321587]: Waiting for error group managing segments... Jan 28 13:15:09.574003214 (309h20m3.057460419s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.574725060 (309h20m3.058182263s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.576637305 (309h20m3.060094506s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.577749105 (309h20m3.061206307s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.578427528 (309h20m3.061884733s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.579417794 (309h20m3.062875s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.592989831 (309h20m3.076447029s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.657755659 (309h20m3.141212856s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.665013100 (309h20m3.148470319s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.670936374 (309h20m3.15439358s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.732123914 (309h20m3.215581118s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:10.114544377 (309h20m3.598001581s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:16:09.396506888 (309h21m2.879964087s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 159.223.143.39:41038->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396531903 (309h21m2.879989091s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: read tcp 159.223.143.39:55138->45.91.116.13:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396681636 (309h21m2.880138835s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12gG7V9H3GkvqC8QmSqmitZaV6xSqR41Rgposb1xnDk8QkXEhFg, address:45.91.116.5:4312): protocol: read tcp 159.223.143.39:36428->45.91.116.5:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396726096 (309h21m2.880183303s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: read tcp 159.223.143.39:47888->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396831206 (309h21m2.880288408s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:1aqBC2ds3GrbQBS5YAnSeJZhH3HMi3DhVGfWw9bvdBUsN32uVx, address:45.91.116.10:4099): protocol: read tcp 159.223.143.39:41914->45.91.116.10:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396984001 (309h21m2.880441211s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 159.223.143.39:55854->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.397024784 (309h21m2.880481982s) [seg_pos={0 0}, upload=321587]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:16:09.416756330 (309h21m2.900213518s) [upload=321587]: Done uploading segments... Jan 28 13:16:09.416792632 (309h21m2.900249829s) [upload=321587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 459.757443ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 519.397231ms trace: https://jaeger.dev.storj.io/trace/25206bcff595778b FAIL (2026-01-28 13:47:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 519.417836ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...XX...X...................................................] duration: 440.464968ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:47:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 440.494164ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 571.714756ms trace: https://jaeger.dev.storj.io/trace/25206bcff595778b FAIL (2026-01-28 13:47:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 571.746501ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 436.306729ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:47:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 436.346854ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 377.307894ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:47:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 377.411396ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 450.54228ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:46:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.572174ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 507.4513ms trace: https://jaeger.dev.storj.io/trace/78b5234b804c56b2 FAIL (2026-01-28 13:46:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (309h20m2.954159537s) [seg_pos={0 0}, upload=321587]: Begin upload segment... Jan 28 13:15:09.470766101 (309h20m2.954223302s) [seg_pos={0 0}, upload=321587]: Done begin upload segment. Jan 28 13:15:09.470798813 (309h20m2.954256012s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471157200 (309h20m2.954614403s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471670574 (309h20m2.955127779s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472180500 (309h20m2.955637705s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472712598 (309h20m2.956169803s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473275419 (309h20m2.956732619s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473668462 (309h20m2.957125659s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474056101 (309h20m2.957513306s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474452312 (309h20m2.957909509s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474782997 (309h20m2.958240194s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.480853378 (309h20m2.964310584s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.481720836 (309h20m2.965178042s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.483099633 (309h20m2.96655683s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.486563577 (309h20m2.970020774s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.492900551 (309h20m2.976357758s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.494672616 (309h20m2.978129817s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.496349386 (309h20m2.979806586s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.498142179 (309h20m2.981599388s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.570661945 (309h20m3.054119149s) [upload=321587]: Next returned nil segment Jan 28 13:15:09.570679413 (309h20m3.05413662s) [upload=321587]: Waiting for error group managing segments... Jan 28 13:15:09.574003214 (309h20m3.057460419s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.574725060 (309h20m3.058182263s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.576637305 (309h20m3.060094506s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.577749105 (309h20m3.061206307s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.578427528 (309h20m3.061884733s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.579417794 (309h20m3.062875s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.592989831 (309h20m3.076447029s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.657755659 (309h20m3.141212856s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.665013100 (309h20m3.148470319s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.670936374 (309h20m3.15439358s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.732123914 (309h20m3.215581118s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:10.114544377 (309h20m3.598001581s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:16:09.396506888 (309h21m2.879964087s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 159.223.143.39:41038->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396531903 (309h21m2.879989091s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: read tcp 159.223.143.39:55138->45.91.116.13:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396681636 (309h21m2.880138835s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12gG7V9H3GkvqC8QmSqmitZaV6xSqR41Rgposb1xnDk8QkXEhFg, address:45.91.116.5:4312): protocol: read tcp 159.223.143.39:36428->45.91.116.5:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396726096 (309h21m2.880183303s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: read tcp 159.223.143.39:47888->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396831206 (309h21m2.880288408s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:1aqBC2ds3GrbQBS5YAnSeJZhH3HMi3DhVGfWw9bvdBUsN32uVx, address:45.91.116.10:4099): protocol: read tcp 159.223.143.39:41914->45.91.116.10:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396984001 (309h21m2.880441211s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 159.223.143.39:55854->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.397024784 (309h21m2.880481982s) [seg_pos={0 0}, upload=321587]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:16:09.416756330 (309h21m2.900213518s) [upload=321587]: Done uploading segments... Jan 28 13:16:09.416792632 (309h21m2.900249829s) [upload=321587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 507.536689ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....XX...X..................................................] duration: 453.956362ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:46:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.979074ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....XX...X.................................................] duration: 492.188364ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:46:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 492.216271ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 513.021005ms trace: https://jaeger.dev.storj.io/trace/25206bcff595778b FAIL (2026-01-28 13:46:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 513.04817ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 440.602462ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:46:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 440.754422ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 491.192941ms trace: https://jaeger.dev.storj.io/trace/78b5234b804c56b2 FAIL (2026-01-28 13:46:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (309h20m2.954159537s) [seg_pos={0 0}, upload=321587]: Begin upload segment... Jan 28 13:15:09.470766101 (309h20m2.954223302s) [seg_pos={0 0}, upload=321587]: Done begin upload segment. Jan 28 13:15:09.470798813 (309h20m2.954256012s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471157200 (309h20m2.954614403s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471670574 (309h20m2.955127779s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472180500 (309h20m2.955637705s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472712598 (309h20m2.956169803s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473275419 (309h20m2.956732619s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473668462 (309h20m2.957125659s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474056101 (309h20m2.957513306s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474452312 (309h20m2.957909509s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474782997 (309h20m2.958240194s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.480853378 (309h20m2.964310584s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.481720836 (309h20m2.965178042s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.483099633 (309h20m2.96655683s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.486563577 (309h20m2.970020774s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.492900551 (309h20m2.976357758s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.494672616 (309h20m2.978129817s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.496349386 (309h20m2.979806586s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.498142179 (309h20m2.981599388s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.570661945 (309h20m3.054119149s) [upload=321587]: Next returned nil segment Jan 28 13:15:09.570679413 (309h20m3.05413662s) [upload=321587]: Waiting for error group managing segments... Jan 28 13:15:09.574003214 (309h20m3.057460419s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.574725060 (309h20m3.058182263s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.576637305 (309h20m3.060094506s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.577749105 (309h20m3.061206307s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.578427528 (309h20m3.061884733s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.579417794 (309h20m3.062875s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.592989831 (309h20m3.076447029s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.657755659 (309h20m3.141212856s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.665013100 (309h20m3.148470319s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.670936374 (309h20m3.15439358s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.732123914 (309h20m3.215581118s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:10.114544377 (309h20m3.598001581s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:16:09.396506888 (309h21m2.879964087s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 159.223.143.39:41038->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396531903 (309h21m2.879989091s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: read tcp 159.223.143.39:55138->45.91.116.13:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396681636 (309h21m2.880138835s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12gG7V9H3GkvqC8QmSqmitZaV6xSqR41Rgposb1xnDk8QkXEhFg, address:45.91.116.5:4312): protocol: read tcp 159.223.143.39:36428->45.91.116.5:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396726096 (309h21m2.880183303s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: read tcp 159.223.143.39:47888->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396831206 (309h21m2.880288408s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:1aqBC2ds3GrbQBS5YAnSeJZhH3HMi3DhVGfWw9bvdBUsN32uVx, address:45.91.116.10:4099): protocol: read tcp 159.223.143.39:41914->45.91.116.10:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396984001 (309h21m2.880441211s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 159.223.143.39:55854->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.397024784 (309h21m2.880481982s) [seg_pos={0 0}, upload=321587]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:16:09.416756330 (309h21m2.900213518s) [upload=321587]: Done uploading segments... Jan 28 13:16:09.416792632 (309h21m2.900249829s) [upload=321587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 491.227324ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 484.495642ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:46:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 484.522209ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 547.167283ms trace: https://jaeger.dev.storj.io/trace/25206bcff595778b FAIL (2026-01-28 13:45:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 547.194218ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 447.87517ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:45:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 447.905289ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 435.488606ms trace: https://jaeger.dev.storj.io/trace/78b5234b804c56b2 FAIL (2026-01-28 13:45:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (309h20m2.954159537s) [seg_pos={0 0}, upload=321587]: Begin upload segment... Jan 28 13:15:09.470766101 (309h20m2.954223302s) [seg_pos={0 0}, upload=321587]: Done begin upload segment. Jan 28 13:15:09.470798813 (309h20m2.954256012s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471157200 (309h20m2.954614403s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471670574 (309h20m2.955127779s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472180500 (309h20m2.955637705s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472712598 (309h20m2.956169803s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473275419 (309h20m2.956732619s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473668462 (309h20m2.957125659s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474056101 (309h20m2.957513306s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474452312 (309h20m2.957909509s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474782997 (309h20m2.958240194s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.480853378 (309h20m2.964310584s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.481720836 (309h20m2.965178042s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.483099633 (309h20m2.96655683s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.486563577 (309h20m2.970020774s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.492900551 (309h20m2.976357758s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.494672616 (309h20m2.978129817s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.496349386 (309h20m2.979806586s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.498142179 (309h20m2.981599388s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.570661945 (309h20m3.054119149s) [upload=321587]: Next returned nil segment Jan 28 13:15:09.570679413 (309h20m3.05413662s) [upload=321587]: Waiting for error group managing segments... Jan 28 13:15:09.574003214 (309h20m3.057460419s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.574725060 (309h20m3.058182263s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.576637305 (309h20m3.060094506s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.577749105 (309h20m3.061206307s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.578427528 (309h20m3.061884733s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.579417794 (309h20m3.062875s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.592989831 (309h20m3.076447029s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.657755659 (309h20m3.141212856s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.665013100 (309h20m3.148470319s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.670936374 (309h20m3.15439358s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.732123914 (309h20m3.215581118s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:10.114544377 (309h20m3.598001581s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:16:09.396506888 (309h21m2.879964087s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 159.223.143.39:41038->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396531903 (309h21m2.879989091s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: read tcp 159.223.143.39:55138->45.91.116.13:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396681636 (309h21m2.880138835s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12gG7V9H3GkvqC8QmSqmitZaV6xSqR41Rgposb1xnDk8QkXEhFg, address:45.91.116.5:4312): protocol: read tcp 159.223.143.39:36428->45.91.116.5:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396726096 (309h21m2.880183303s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: read tcp 159.223.143.39:47888->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396831206 (309h21m2.880288408s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:1aqBC2ds3GrbQBS5YAnSeJZhH3HMi3DhVGfWw9bvdBUsN32uVx, address:45.91.116.10:4099): protocol: read tcp 159.223.143.39:41914->45.91.116.10:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396984001 (309h21m2.880441211s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 159.223.143.39:55854->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.397024784 (309h21m2.880481982s) [seg_pos={0 0}, upload=321587]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:16:09.416756330 (309h21m2.900213518s) [upload=321587]: Done uploading segments... Jan 28 13:16:09.416792632 (309h21m2.900249829s) [upload=321587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 435.526189ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 515.082838ms trace: https://jaeger.dev.storj.io/trace/25206bcff595778b FAIL (2026-01-28 13:45:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 515.119205ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 432.285112ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:45:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 432.314861ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......XX...X................................................] duration: 451.400217ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:45:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 451.459972ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 560.283272ms trace: https://jaeger.dev.storj.io/trace/78b5234b804c56b2 FAIL (2026-01-28 13:45:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (309h20m2.954159537s) [seg_pos={0 0}, upload=321587]: Begin upload segment... Jan 28 13:15:09.470766101 (309h20m2.954223302s) [seg_pos={0 0}, upload=321587]: Done begin upload segment. Jan 28 13:15:09.470798813 (309h20m2.954256012s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471157200 (309h20m2.954614403s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471670574 (309h20m2.955127779s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472180500 (309h20m2.955637705s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472712598 (309h20m2.956169803s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473275419 (309h20m2.956732619s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473668462 (309h20m2.957125659s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474056101 (309h20m2.957513306s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474452312 (309h20m2.957909509s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474782997 (309h20m2.958240194s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.480853378 (309h20m2.964310584s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.481720836 (309h20m2.965178042s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.483099633 (309h20m2.96655683s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.486563577 (309h20m2.970020774s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.492900551 (309h20m2.976357758s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.494672616 (309h20m2.978129817s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.496349386 (309h20m2.979806586s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.498142179 (309h20m2.981599388s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.570661945 (309h20m3.054119149s) [upload=321587]: Next returned nil segment Jan 28 13:15:09.570679413 (309h20m3.05413662s) [upload=321587]: Waiting for error group managing segments... Jan 28 13:15:09.574003214 (309h20m3.057460419s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.574725060 (309h20m3.058182263s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.576637305 (309h20m3.060094506s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.577749105 (309h20m3.061206307s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.578427528 (309h20m3.061884733s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.579417794 (309h20m3.062875s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.592989831 (309h20m3.076447029s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.657755659 (309h20m3.141212856s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.665013100 (309h20m3.148470319s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.670936374 (309h20m3.15439358s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.732123914 (309h20m3.215581118s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:10.114544377 (309h20m3.598001581s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:16:09.396506888 (309h21m2.879964087s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 159.223.143.39:41038->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396531903 (309h21m2.879989091s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: read tcp 159.223.143.39:55138->45.91.116.13:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396681636 (309h21m2.880138835s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12gG7V9H3GkvqC8QmSqmitZaV6xSqR41Rgposb1xnDk8QkXEhFg, address:45.91.116.5:4312): protocol: read tcp 159.223.143.39:36428->45.91.116.5:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396726096 (309h21m2.880183303s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: read tcp 159.223.143.39:47888->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396831206 (309h21m2.880288408s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:1aqBC2ds3GrbQBS5YAnSeJZhH3HMi3DhVGfWw9bvdBUsN32uVx, address:45.91.116.10:4099): protocol: read tcp 159.223.143.39:41914->45.91.116.10:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396984001 (309h21m2.880441211s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 159.223.143.39:55854->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.397024784 (309h21m2.880481982s) [seg_pos={0 0}, upload=321587]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:16:09.416756330 (309h21m2.900213518s) [upload=321587]: Done uploading segments... Jan 28 13:16:09.416792632 (309h21m2.900249829s) [upload=321587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 560.306443ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 459.823663ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:44:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 459.850579ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 396.23574ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:44:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 396.325877ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 415.307831ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:44:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.338478ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 442.127716ms trace: https://jaeger.dev.storj.io/trace/78b5234b804c56b2 FAIL (2026-01-28 13:44:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (309h20m2.954159537s) [seg_pos={0 0}, upload=321587]: Begin upload segment... Jan 28 13:15:09.470766101 (309h20m2.954223302s) [seg_pos={0 0}, upload=321587]: Done begin upload segment. Jan 28 13:15:09.470798813 (309h20m2.954256012s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471157200 (309h20m2.954614403s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471670574 (309h20m2.955127779s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472180500 (309h20m2.955637705s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472712598 (309h20m2.956169803s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473275419 (309h20m2.956732619s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473668462 (309h20m2.957125659s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474056101 (309h20m2.957513306s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474452312 (309h20m2.957909509s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474782997 (309h20m2.958240194s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.480853378 (309h20m2.964310584s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.481720836 (309h20m2.965178042s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.483099633 (309h20m2.96655683s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.486563577 (309h20m2.970020774s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.492900551 (309h20m2.976357758s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.494672616 (309h20m2.978129817s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.496349386 (309h20m2.979806586s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.498142179 (309h20m2.981599388s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.570661945 (309h20m3.054119149s) [upload=321587]: Next returned nil segment Jan 28 13:15:09.570679413 (309h20m3.05413662s) [upload=321587]: Waiting for error group managing segments... Jan 28 13:15:09.574003214 (309h20m3.057460419s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.574725060 (309h20m3.058182263s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.576637305 (309h20m3.060094506s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.577749105 (309h20m3.061206307s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.578427528 (309h20m3.061884733s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.579417794 (309h20m3.062875s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.592989831 (309h20m3.076447029s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.657755659 (309h20m3.141212856s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.665013100 (309h20m3.148470319s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.670936374 (309h20m3.15439358s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.732123914 (309h20m3.215581118s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:10.114544377 (309h20m3.598001581s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:16:09.396506888 (309h21m2.879964087s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 159.223.143.39:41038->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396531903 (309h21m2.879989091s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: read tcp 159.223.143.39:55138->45.91.116.13:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396681636 (309h21m2.880138835s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12gG7V9H3GkvqC8QmSqmitZaV6xSqR41Rgposb1xnDk8QkXEhFg, address:45.91.116.5:4312): protocol: read tcp 159.223.143.39:36428->45.91.116.5:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396726096 (309h21m2.880183303s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: read tcp 159.223.143.39:47888->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396831206 (309h21m2.880288408s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:1aqBC2ds3GrbQBS5YAnSeJZhH3HMi3DhVGfWw9bvdBUsN32uVx, address:45.91.116.10:4099): protocol: read tcp 159.223.143.39:41914->45.91.116.10:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396984001 (309h21m2.880441211s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 159.223.143.39:55854->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.397024784 (309h21m2.880481982s) [seg_pos={0 0}, upload=321587]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:16:09.416756330 (309h21m2.900213518s) [upload=321587]: Done uploading segments... Jan 28 13:16:09.416792632 (309h21m2.900249829s) [upload=321587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 442.15959ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XX...X...............................................] duration: 448.986558ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:44:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 449.016594ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 511.174541ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:44:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 511.205096ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 618.148748ms trace: https://jaeger.dev.storj.io/trace/25206bcff595778b FAIL (2026-01-28 13:44:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 618.255602ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 470.545979ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:43:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.584202ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 447.810438ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:43:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 447.984071ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 503.683924ms trace: https://jaeger.dev.storj.io/trace/78b5234b804c56b2 FAIL (2026-01-28 13:43:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (309h20m2.954159537s) [seg_pos={0 0}, upload=321587]: Begin upload segment... Jan 28 13:15:09.470766101 (309h20m2.954223302s) [seg_pos={0 0}, upload=321587]: Done begin upload segment. Jan 28 13:15:09.470798813 (309h20m2.954256012s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471157200 (309h20m2.954614403s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471670574 (309h20m2.955127779s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472180500 (309h20m2.955637705s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472712598 (309h20m2.956169803s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473275419 (309h20m2.956732619s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473668462 (309h20m2.957125659s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474056101 (309h20m2.957513306s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474452312 (309h20m2.957909509s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474782997 (309h20m2.958240194s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.480853378 (309h20m2.964310584s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.481720836 (309h20m2.965178042s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.483099633 (309h20m2.96655683s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.486563577 (309h20m2.970020774s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.492900551 (309h20m2.976357758s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.494672616 (309h20m2.978129817s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.496349386 (309h20m2.979806586s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.498142179 (309h20m2.981599388s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.570661945 (309h20m3.054119149s) [upload=321587]: Next returned nil segment Jan 28 13:15:09.570679413 (309h20m3.05413662s) [upload=321587]: Waiting for error group managing segments... Jan 28 13:15:09.574003214 (309h20m3.057460419s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.574725060 (309h20m3.058182263s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.576637305 (309h20m3.060094506s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.577749105 (309h20m3.061206307s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.578427528 (309h20m3.061884733s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.579417794 (309h20m3.062875s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.592989831 (309h20m3.076447029s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.657755659 (309h20m3.141212856s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.665013100 (309h20m3.148470319s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.670936374 (309h20m3.15439358s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.732123914 (309h20m3.215581118s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:10.114544377 (309h20m3.598001581s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:16:09.396506888 (309h21m2.879964087s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 159.223.143.39:41038->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396531903 (309h21m2.879989091s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: read tcp 159.223.143.39:55138->45.91.116.13:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396681636 (309h21m2.880138835s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12gG7V9H3GkvqC8QmSqmitZaV6xSqR41Rgposb1xnDk8QkXEhFg, address:45.91.116.5:4312): protocol: read tcp 159.223.143.39:36428->45.91.116.5:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396726096 (309h21m2.880183303s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: read tcp 159.223.143.39:47888->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396831206 (309h21m2.880288408s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:1aqBC2ds3GrbQBS5YAnSeJZhH3HMi3DhVGfWw9bvdBUsN32uVx, address:45.91.116.10:4099): protocol: read tcp 159.223.143.39:41914->45.91.116.10:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396984001 (309h21m2.880441211s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 159.223.143.39:55854->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.397024784 (309h21m2.880481982s) [seg_pos={0 0}, upload=321587]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:16:09.416756330 (309h21m2.900213518s) [upload=321587]: Done uploading segments... Jan 28 13:16:09.416792632 (309h21m2.900249829s) [upload=321587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 503.785858ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 466.728519ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:43:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.749744ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........XX...X..............................................] duration: 468.245939ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:43:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 468.359567ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 457.105282ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:43:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 457.130458ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 621.350914ms trace: https://jaeger.dev.storj.io/trace/25206bcff595778b FAIL (2026-01-28 13:42:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 621.38543ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 442.107298ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:42:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.148229ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 401.545556ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:42:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 401.579337ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 501.235296ms trace: https://jaeger.dev.storj.io/trace/78b5234b804c56b2 FAIL (2026-01-28 13:42:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (309h20m2.954159537s) [seg_pos={0 0}, upload=321587]: Begin upload segment... Jan 28 13:15:09.470766101 (309h20m2.954223302s) [seg_pos={0 0}, upload=321587]: Done begin upload segment. Jan 28 13:15:09.470798813 (309h20m2.954256012s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471157200 (309h20m2.954614403s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471670574 (309h20m2.955127779s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472180500 (309h20m2.955637705s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472712598 (309h20m2.956169803s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473275419 (309h20m2.956732619s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473668462 (309h20m2.957125659s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474056101 (309h20m2.957513306s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474452312 (309h20m2.957909509s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474782997 (309h20m2.958240194s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.480853378 (309h20m2.964310584s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.481720836 (309h20m2.965178042s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.483099633 (309h20m2.96655683s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.486563577 (309h20m2.970020774s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.492900551 (309h20m2.976357758s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.494672616 (309h20m2.978129817s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.496349386 (309h20m2.979806586s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.498142179 (309h20m2.981599388s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.570661945 (309h20m3.054119149s) [upload=321587]: Next returned nil segment Jan 28 13:15:09.570679413 (309h20m3.05413662s) [upload=321587]: Waiting for error group managing segments... Jan 28 13:15:09.574003214 (309h20m3.057460419s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.574725060 (309h20m3.058182263s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.576637305 (309h20m3.060094506s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.577749105 (309h20m3.061206307s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.578427528 (309h20m3.061884733s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.579417794 (309h20m3.062875s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.592989831 (309h20m3.076447029s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.657755659 (309h20m3.141212856s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.665013100 (309h20m3.148470319s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.670936374 (309h20m3.15439358s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.732123914 (309h20m3.215581118s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:10.114544377 (309h20m3.598001581s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:16:09.396506888 (309h21m2.879964087s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 159.223.143.39:41038->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396531903 (309h21m2.879989091s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: read tcp 159.223.143.39:55138->45.91.116.13:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396681636 (309h21m2.880138835s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12gG7V9H3GkvqC8QmSqmitZaV6xSqR41Rgposb1xnDk8QkXEhFg, address:45.91.116.5:4312): protocol: read tcp 159.223.143.39:36428->45.91.116.5:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396726096 (309h21m2.880183303s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: read tcp 159.223.143.39:47888->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396831206 (309h21m2.880288408s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:1aqBC2ds3GrbQBS5YAnSeJZhH3HMi3DhVGfWw9bvdBUsN32uVx, address:45.91.116.10:4099): protocol: read tcp 159.223.143.39:41914->45.91.116.10:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396984001 (309h21m2.880441211s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 159.223.143.39:55854->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.397024784 (309h21m2.880481982s) [seg_pos={0 0}, upload=321587]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:16:09.416756330 (309h21m2.900213518s) [upload=321587]: Done uploading segments... Jan 28 13:16:09.416792632 (309h21m2.900249829s) [upload=321587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 501.305065ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........XX...X.............................................] duration: 472.359224ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:42:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 472.392746ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 439.398094ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:42:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.500163ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 501.156402ms trace: https://jaeger.dev.storj.io/trace/78b5234b804c56b2 FAIL (2026-01-28 13:42:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (309h20m2.954159537s) [seg_pos={0 0}, upload=321587]: Begin upload segment... Jan 28 13:15:09.470766101 (309h20m2.954223302s) [seg_pos={0 0}, upload=321587]: Done begin upload segment. Jan 28 13:15:09.470798813 (309h20m2.954256012s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471157200 (309h20m2.954614403s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471670574 (309h20m2.955127779s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472180500 (309h20m2.955637705s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472712598 (309h20m2.956169803s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473275419 (309h20m2.956732619s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473668462 (309h20m2.957125659s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474056101 (309h20m2.957513306s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474452312 (309h20m2.957909509s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474782997 (309h20m2.958240194s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.480853378 (309h20m2.964310584s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.481720836 (309h20m2.965178042s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.483099633 (309h20m2.96655683s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.486563577 (309h20m2.970020774s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.492900551 (309h20m2.976357758s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.494672616 (309h20m2.978129817s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.496349386 (309h20m2.979806586s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.498142179 (309h20m2.981599388s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.570661945 (309h20m3.054119149s) [upload=321587]: Next returned nil segment Jan 28 13:15:09.570679413 (309h20m3.05413662s) [upload=321587]: Waiting for error group managing segments... Jan 28 13:15:09.574003214 (309h20m3.057460419s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.574725060 (309h20m3.058182263s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.576637305 (309h20m3.060094506s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.577749105 (309h20m3.061206307s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.578427528 (309h20m3.061884733s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.579417794 (309h20m3.062875s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.592989831 (309h20m3.076447029s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.657755659 (309h20m3.141212856s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.665013100 (309h20m3.148470319s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.670936374 (309h20m3.15439358s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.732123914 (309h20m3.215581118s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:10.114544377 (309h20m3.598001581s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:16:09.396506888 (309h21m2.879964087s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 159.223.143.39:41038->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396531903 (309h21m2.879989091s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: read tcp 159.223.143.39:55138->45.91.116.13:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396681636 (309h21m2.880138835s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12gG7V9H3GkvqC8QmSqmitZaV6xSqR41Rgposb1xnDk8QkXEhFg, address:45.91.116.5:4312): protocol: read tcp 159.223.143.39:36428->45.91.116.5:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396726096 (309h21m2.880183303s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: read tcp 159.223.143.39:47888->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396831206 (309h21m2.880288408s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:1aqBC2ds3GrbQBS5YAnSeJZhH3HMi3DhVGfWw9bvdBUsN32uVx, address:45.91.116.10:4099): protocol: read tcp 159.223.143.39:41914->45.91.116.10:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396984001 (309h21m2.880441211s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 159.223.143.39:55854->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.397024784 (309h21m2.880481982s) [seg_pos={0 0}, upload=321587]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:16:09.416756330 (309h21m2.900213518s) [upload=321587]: Done uploading segments... Jan 28 13:16:09.416792632 (309h21m2.900249829s) [upload=321587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 501.252317ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 580.903192ms trace: https://jaeger.dev.storj.io/trace/25206bcff595778b FAIL (2026-01-28 13:41:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 580.93422ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 390.492464ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:41:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 390.517692ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........XX...X............................................] duration: 462.252738ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:41:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 462.286354ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........XX...X...........................................] duration: 535.01703ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:41:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 535.100773ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 428.55134ms trace: https://jaeger.dev.storj.io/trace/29d17239f2c5e4cb FAIL (2026-01-28 13:41:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Jan 28 13:06:03.887182974 (309h10m57.370640178s) [upload=321438]: Starting upload Jan 28 13:06:03.887271177 (309h10m57.370728382s) [upload=321438]: Done starting upload Jan 28 13:06:03.887509913 (309h10m57.37096711s) [upload=321438]: Uploading segments... Jan 28 13:06:03.887731838 (309h10m57.371189042s) [upload=321438]: Got next segment. Inline: false Jan 28 13:06:03.887815992 (309h10m57.37127318s) [upload=321438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:06:03.887822935 (309h10m57.371280124s) [upload=321438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:06:03.914873804 (309h10m57.398331011s) [seg_pos={0 0}, upload=321438]: Begin upload segment... Jan 28 13:06:03.914949492 (309h10m57.398406692s) [seg_pos={0 0}, upload=321438]: Done begin upload segment. Jan 28 13:06:03.915002521 (309h10m57.398459709s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915050818 (309h10m57.398508025s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.915699007 (309h10m57.399156215s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916030449 (309h10m57.399487656s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.916490205 (309h10m57.399947411s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917055473 (309h10m57.40051268s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917502447 (309h10m57.400959656s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.917902811 (309h10m57.401360252s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918310462 (309h10m57.401767669s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.918705222 (309h10m57.402162434s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919225383 (309h10m57.402682608s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919484326 (309h10m57.402941536s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919763594 (309h10m57.403220804s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.919996930 (309h10m57.403454138s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.920565490 (309h10m57.404022696s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921288098 (309h10m57.404745307s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.921890836 (309h10m57.405348039s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.922435477 (309h10m57.405892682s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Uploading piece... Jan 28 13:06:03.993917367 (309h10m57.477374558s) [upload=321438]: Next returned nil segment Jan 28 13:06:03.993933751 (309h10m57.477390951s) [upload=321438]: Waiting for error group managing segments... Jan 28 13:06:03.998003627 (309h10m57.481460824s) [noise=true , address=77.90.31.16:4741 , piece_id=SSPCWLHORPXMTDVY2MJMKTII2LF2BGMCGEOAZV76TPYV6VCP4UQQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000059268 (309h10m57.483516496s) [noise=true , address=77.90.31.18:4813 , piece_id=UPOBUAQIVCBAYYRTA2TGDQAUO4GZTEO342TCOUOB2MUYZP64XERQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000514555 (309h10m57.483971745s) [noise=true , address=79.127.219.41:4931 , piece_id=PGEOXKF4J6SGSJRJPVEDTSLSFQGRX7D2UNHPBLW6N7IOZUEIRVJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.000708614 (309h10m57.48416584s) [noise=true , address=79.127.219.38:4820 , piece_id=RHD3PAOTHC4ENLTQ27STXVKJNYTO3AJCHPPGOPLOCVDP24SYVISQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001640447 (309h10m57.485097676s) [noise=true , address=79.127.219.45:4875 , piece_id=CNY5KE67VXYZEPG6E2X3IT36YONGSHLDPT67ZHMWZ7657DARWNZA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.001894328 (309h10m57.485351546s) [noise=true , address=79.127.219.46:4889 , piece_id=442MSKPHATJWYCJB55R5INX4YRYMFSXIQ2WBVI72IKWJWRLLS7BA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.002391772 (309h10m57.48584897s) [noise=true , address=145.79.227.52:4461 , piece_id=CGDBDRACQXVJLADEZ6EL6623PO64MHRN6IK6DO5M764QLGNJW2BQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.017070746 (309h10m57.500527953s) [noise=true , address=77.90.31.14:4671 , piece_id=K3HIVN2ZV3PVGTDFVPLLTM6PBLFTAS7MWX4HLDO3YIR532P7KIFA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.036495818 (309h10m57.519953018s) [noise=true , address=77.90.31.12:4604 , piece_id=X67RII7QBWX3G6PYNZ67NFR2BSTAEOFHACEU4TTDHSEJESAD73NQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:04.516564908 (309h10m58.000022117s) [noise=true , address=64.49.13.169:4714 , piece_id=6JVO4NFMHWCV5LIYHP6TAC3KO7MCD3IRI66H7LB46IL4KHVNEUGQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:06.062633632 (309h10m59.546090825s) [noise=true , address=64.49.13.169:4706 , piece_id=YY6VDZ2744CJ5ZYTHVPYHG5GKP6X3FZGH5MHEBOJ3EA3D7LIUNLQ, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:06:45.470837617 (309h11m38.954294815s) [noise=true , address=45.91.116.7:4033 , piece_id=X26B4XYBHG4NWD4T7HKU2RT352O6G4SMEIMSGIIPA54GA7PG5ZQA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: <nil> Jan 28 13:07:03.839781411 (309h11m57.323238613s) [noise=true , address=45.91.116.9:4273 , piece_id=PXCASBW4LMN5LQLTZMINZN6HVUNBKTATOEX7KGY4FHEPRNSOIYNA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1u1eZxqD3miaYLw3ZyK6EvRhpYq9sfkgbVVFGnqEPTggVgQUNf, address:45.91.116.9:4273): protocol: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58322->45.91.116.9:4273: use of closed network connection Jan 28 13:07:03.839870815 (309h11m57.323328052s) [noise=true , address=45.91.116.6:4983 , piece_id=AXGYGJZG7QW46T6JLEJSVLYHZJFOHUAABE2N4NBR5AEFFOEXZKJA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: rpc: dial tcp 45.91.116.6:4983: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.839922555 (309h11m57.323379754s) [noise=true , address=45.91.116.4:4970 , piece_id=RTMOQWA55GLGYCJ55LYLLTE6VNLNIHHXO5PQSBBW35E3TOAWKWPA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe, address:45.91.116.4:4970): protocol: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:42472->45.91.116.4:4970: use of closed network connection Jan 28 13:07:03.840151963 (309h11m57.323609175s) [noise=true , address=45.91.116.5:4307 , piece_id=NRDDOQSPMVZTA43IJUZY7CP2LDRW7OTPG7XBMRMXUEALRZZU5IAA, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: write tcp 159.223.143.39:60786->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.840372333 (309h11m57.323829539s) [noise=true , address=45.91.116.14:4195 , piece_id=SFDZ5DKSL6TLOQJBQDWLNNPY4AJ3KAJUZX6XDHQAKH2NZRYNQK5Q, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:12N8gL2cPKtuvwn66hcZWQsoe3KRUYgHoy5FTJ38KZstPzzxiKt, address:45.91.116.14:4195): protocol: rpc: dial tcp 45.91.116.14:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841055183 (309h11m57.324512389s) [noise=true , address=45.91.116.12:4259 , piece_id=4WA2KXBR4FDJFX7647AFGNJN3FUXFS6TTRKLWNHIDKMKWZMAYL4A, seg_pos={0 0}, upload=321438]: Done uploading piece. err: ecclient: upload failed (node:124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, address:45.91.116.12:4259): protocol: rpc: dial tcp 45.91.116.12:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:07:03.841113008 (309h11m57.324570198s) [seg_pos={0 0}, upload=321438]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:07:03.871592717 (309h11m57.35505014s) [upload=321438]: Done uploading segments... Jan 28 13:07:03.871638212 (309h11m57.35509542s) [upload=321438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 428.585925ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 474.485968ms trace: https://jaeger.dev.storj.io/trace/2335c576b739e067 FAIL (2026-01-28 13:40:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Jan 28 13:09:55.596658066 (309h14m49.080115276s) [upload=321503]: Starting upload Jan 28 13:09:55.596804643 (309h14m49.080261882s) [upload=321503]: Done starting upload Jan 28 13:09:55.597514335 (309h14m49.080971534s) [upload=321503]: Uploading segments... Jan 28 13:09:55.597625932 (309h14m49.081083136s) [upload=321503]: Got next segment. Inline: false Jan 28 13:09:55.597690430 (309h14m49.081147635s) [upload=321503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 28 13:09:55.597697562 (309h14m49.081154799s) [upload=321503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 28 13:09:55.626540179 (309h14m49.109997384s) [seg_pos={0 0}, upload=321503]: Begin upload segment... Jan 28 13:09:55.626663888 (309h14m49.110121089s) [seg_pos={0 0}, upload=321503]: Done begin upload segment. Jan 28 13:09:55.626708863 (309h14m49.110166073s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627141738 (309h14m49.110598947s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627876363 (309h14m49.111333565s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.627896924 (309h14m49.111354124s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628347627 (309h14m49.111804824s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628417238 (309h14m49.111874449s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628774720 (309h14m49.112231917s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.628972749 (309h14m49.112429957s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629305197 (309h14m49.112762397s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629464750 (309h14m49.11292195s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629872321 (309h14m49.113329529s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.629882180 (309h14m49.113339379s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.630506941 (309h14m49.113964149s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631157842 (309h14m49.114615046s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.631890406 (309h14m49.115347613s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.634559748 (309h14m49.118016932s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Uploading piece... Jan 28 13:09:55.693935355 (309h14m49.177392538s) [upload=321503]: Next returned nil segment Jan 28 13:09:55.693948540 (309h14m49.177405741s) [upload=321503]: Waiting for error group managing segments... Jan 28 13:09:55.704566602 (309h14m49.188023804s) [noise=true , address=77.90.31.15:4700 , piece_id=RXW7CCU2W4OKJTIOXPUT7JVTM2WSE6YTEALBC7ZLJP4EX5S675SA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707168214 (309h14m49.190625403s) [noise=true , address=145.79.227.43:4158 , piece_id=NCCBNK6PRWUBK2QGUHT6BUF762M3KNWIB5FA6OBQWN2VMGEQPKIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.707224148 (309h14m49.190681348s) [noise=true , address=77.90.31.17:4770 , piece_id=N44E4JVQ2SF5QL6TIR3I7FZUF6MPWCXHUAWHNJSP5FML4IY6ZTZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.708576850 (309h14m49.19203405s) [noise=true , address=145.79.227.40:4049 , piece_id=PQIBWPVLSOGVQA6V2NNC4FHTODVUT63YQAZQXCPOXMP6VTM6LJDQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712022077 (309h14m49.195479264s) [noise=true , address=145.79.227.38:4979 , piece_id=JXJYJG2KC6KVAS32EJZR7LOSXP5SR33YF2S6U7IEUNE7NFZGUWVA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712239097 (309h14m49.195696306s) [noise=true , address=145.79.227.52:4446 , piece_id=AZF4WRVFPQPH66X25LJKK65MDVRIDQPIA7W4DHDI5YEIS56PMYBQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.712589926 (309h14m49.196047124s) [noise=true , address=145.79.227.29:4942 , piece_id=BSUAEH3OY4XVNZ6U56EWZLRIGMVP4E2Y7DH2II5BDNLP2GA25MKQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713025197 (309h14m49.196482404s) [noise=true , address=145.79.227.31:4799 , piece_id=4ENOPIOJPIMQTRWSGOLHG4PMEBQAK3ASYBMLI4T4M7YOTUE3PFSA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.713790182 (309h14m49.197247369s) [noise=true , address=145.79.227.50:4375 , piece_id=IRSUAW3GZXKL4RCQHWVEE4SZUJT2EZ4UV2LAF443JUUEU6UVNVJA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.715409721 (309h14m49.198866958s) [noise=true , address=145.79.227.11:4729 , piece_id=PZYTKOLE64KBK62MNGKOYJGJ6OGAWAJ2UE3MTL4IORTYNZS2BCIQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.746028202 (309h14m49.2294854s) [noise=true , address=77.90.31.10:4517 , piece_id=Q2DZJJAF35HJC32DWECM5G2PK37J7HR3TCFW3YQS3QAMAAFPCWFA, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:55.749254547 (309h14m49.232711747s) [noise=true , address=77.90.31.11:4555 , piece_id=MLN2EH363CNFD2FLC6SOXARPMADZNCFCKBKQDLZSW274WXUO2XLQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: <nil> Jan 28 13:09:56.052159318 (309h14m49.535616525s) [noise=true , address=145.79.227.36:4519 , piece_id=PPDLP67CHWD4OAJFSNAKPZF264ZSP5OYICVMEUG5UJBKSSOGAIWQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:145.79.227.36:4519): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/20/log-0000000000008620-00000000: read-only file system; EOF Jan 28 13:10:25.559895927 (309h15m19.043353124s) [noise=true , address=45.91.116.5:4298 , piece_id=3MSXTZOMEGHO47BK7OU4M6WEUGT77BPK5QOHKM4SWJX6IFEKOEZQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 159.223.143.39:51832->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560162960 (309h15m19.043620157s) [noise=true , address=45.91.116.12:4228 , piece_id=VFQOETZUQXDADC5EAUZOHIEOGDYD4XKLNKKPEKVFK4GMX5NJQ62Q, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, address:45.91.116.12:4228): protocol: read tcp 159.223.143.39:37408->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:10:25.560807954 (309h15m19.044265157s) [noise=true , address=45.91.116.13:4155 , piece_id=TEKN4SAI3PASYB7JZW54S7NFO3N7VVGBB7UJTFGEYUYRI7PHWBOQ, seg_pos={0 0}, upload=321503]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: context deadline exceeded; context deadline exceeded Jan 28 13:10:25.560972098 (309h15m19.044429283s) [seg_pos={0 0}, upload=321503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jan 28 13:10:25.584545069 (309h15m19.068002275s) [upload=321503]: Done uploading segments... Jan 28 13:10:25.584606650 (309h15m19.068063851s) [upload=321503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 474.509754ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 456.966869ms trace: https://jaeger.dev.storj.io/trace/78b5234b804c56b2 FAIL (2026-01-28 13:40:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (309h20m2.954159537s) [seg_pos={0 0}, upload=321587]: Begin upload segment... Jan 28 13:15:09.470766101 (309h20m2.954223302s) [seg_pos={0 0}, upload=321587]: Done begin upload segment. Jan 28 13:15:09.470798813 (309h20m2.954256012s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471157200 (309h20m2.954614403s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.471670574 (309h20m2.955127779s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472180500 (309h20m2.955637705s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.472712598 (309h20m2.956169803s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473275419 (309h20m2.956732619s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.473668462 (309h20m2.957125659s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474056101 (309h20m2.957513306s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474452312 (309h20m2.957909509s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.474782997 (309h20m2.958240194s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.480853378 (309h20m2.964310584s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.481720836 (309h20m2.965178042s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.483099633 (309h20m2.96655683s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.486563577 (309h20m2.970020774s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.492900551 (309h20m2.976357758s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.494672616 (309h20m2.978129817s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.496349386 (309h20m2.979806586s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.498142179 (309h20m2.981599388s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Uploading piece... Jan 28 13:15:09.570661945 (309h20m3.054119149s) [upload=321587]: Next returned nil segment Jan 28 13:15:09.570679413 (309h20m3.05413662s) [upload=321587]: Waiting for error group managing segments... Jan 28 13:15:09.574003214 (309h20m3.057460419s) [noise=true , address=79.127.219.38:4845 , piece_id=DCJUX2V4SXQ6E66QP2LAPXHHYDO3SIQVW7FQ5QCHACBHHSEISCXQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.574725060 (309h20m3.058182263s) [noise=true , address=79.127.219.45:4853 , piece_id=CUMLY6TQLXI2M5THYPSMCLDWKJ5VEMZAC4NDWTLAKRH4LK53BX3Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.576637305 (309h20m3.060094506s) [noise=true , address=79.127.219.41:4941 , piece_id=XJQWVY3CISOR2JSRDFCZFVW2DQDU5XAWE63RIP7QPVMVZNL7AH2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.577749105 (309h20m3.061206307s) [noise=true , address=77.90.31.19:4848 , piece_id=EOXCDQWQCUB4EVLXN2VTJIQGWNA4HQEWYBDWMLWY7PKGMDDPJ63Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.578427528 (309h20m3.061884733s) [noise=true , address=145.79.227.17:4998 , piece_id=BXODTPFA5FF7UP46IUVP4F3PMQP6LYYBMYUZQIFQRALKWKRNMKDQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.579417794 (309h20m3.062875s) [noise=true , address=145.79.227.52:4476 , piece_id=CWDQ2W2NONHKK6JERTME4AMNLHE3NLM36MPNVYC7PDZML66HVEZA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.592989831 (309h20m3.076447029s) [noise=true , address=77.90.31.18:4834 , piece_id=ZY6FZGVLMFJQP3C7ZOXK2L5ZK3JQ36QURSRAZUZFB7OCDC2NDG2Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.657755659 (309h20m3.141212856s) [noise=true , address=77.90.31.13:4649 , piece_id=IXZFAN4X5XUBEGC4MO6TNGKS4EZGLI2TKIO5MZV6E35OX2KZBSGA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.665013100 (309h20m3.148470319s) [noise=true , address=77.90.31.14:4684 , piece_id=UAMIB7W4453YQEMTWZCUWSGDF6CFFQUBNM3E57MBOUWUNKC2IZBA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.670936374 (309h20m3.15439358s) [noise=true , address=77.90.31.17:4797 , piece_id=MZC4SYBBDAVINPWGAFPUDWHC7QIQC23GARHGEUGFCZYV3KKSCLPA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:09.732123914 (309h20m3.215581118s) [noise=true , address=77.90.31.15:4729 , piece_id=ZCCV4RS5675KJNIQO5CWN7RBFSUVYQTQXXWUU77OXX3XBYVSBUSQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:15:10.114544377 (309h20m3.598001581s) [noise=true , address=64.49.13.138:4490 , piece_id=GNWB26QFLJRUCDJUIR4WAHRWGA46OZXNHKDDYEDDSJPETOQIIEMA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: <nil> Jan 28 13:16:09.396506888 (309h21m2.879964087s) [noise=true , address=45.91.116.11:4142 , piece_id=4V7LELZYACY362P433HTY6ZCNYRCWFMQGRVINGIPG65VMRI23DRQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 159.223.143.39:41038->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396531903 (309h21m2.879989091s) [noise=true , address=45.91.116.13:4159 , piece_id=425A66QNGRTDKN6JGHIBG5A3BHUBRBVAMUVJULIAVJPERFX55YRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: read tcp 159.223.143.39:55138->45.91.116.13:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396681636 (309h21m2.880138835s) [noise=true , address=45.91.116.5:4312 , piece_id=K2AVPA4WBWTXMD2DKCEY3NKRRUFNN2OGV2KKZ7SNRJFXSPAXPSRA, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12gG7V9H3GkvqC8QmSqmitZaV6xSqR41Rgposb1xnDk8QkXEhFg, address:45.91.116.5:4312): protocol: read tcp 159.223.143.39:36428->45.91.116.5:4312: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396726096 (309h21m2.880183303s) [noise=true , address=45.91.116.14:4193 , piece_id=JGLWWWDYRTAV3LBX53POHHYVVN7AUORCB7WJX6J4DOBVE37PJBWQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: read tcp 159.223.143.39:47888->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396831206 (309h21m2.880288408s) [noise=true , address=45.91.116.10:4099 , piece_id=S6XSDVAWOJYEYRNUCHNKCY4EO6EJQI2XRFW7ELRFAVGXCQLM7SLQ, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:1aqBC2ds3GrbQBS5YAnSeJZhH3HMi3DhVGfWw9bvdBUsN32uVx, address:45.91.116.10:4099): protocol: read tcp 159.223.143.39:41914->45.91.116.10:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.396984001 (309h21m2.880441211s) [noise=true , address=45.91.116.4:4962 , piece_id=E3MFHRHLVRLT6FOF6GABAJBHUY2VWQDXXQFGKAKKSVZCV6QTZ34Q, seg_pos={0 0}, upload=321587]: Done uploading piece. err: ecclient: upload failed (node:115Q9FLfn9BcC5kHcdAWa7yn9myRVusdj12Qcy7nvLr1kUXdvF, address:45.91.116.4:4962): protocol: read tcp 159.223.143.39:55854->45.91.116.4:4962: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 28 13:16:09.397024784 (309h21m2.880481982s) [seg_pos={0 0}, upload=321587]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jan 28 13:16:09.416756330 (309h21m2.900213518s) [upload=321587]: Done uploading segments... Jan 28 13:16:09.416792632 (309h21m2.900249829s) [upload=321587]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 457.059527ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XX...X..........................................] duration: 479.878621ms trace: https://jaeger.dev.storj.io/trace/2b19f5796d2a9547 FAIL (2026-01-28 13:40:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.911408ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)