Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [.......................................X......X......XX.X...] duration: 42.791199055s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:19:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 42.791231255s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X...X...X...................X......X...........] duration: 1.138550728s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:19:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.138576981s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 631.007838ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:19:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 631.032713ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.......X.............X....X..........X...............] duration: 1.284361649s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:19:19 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.284390038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 678.71788ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:19:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 678.742605ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [........................................X......X......XX.X..] duration: 16.359030354s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:18:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 16.35905786s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X...X...X...................X......X..........] duration: 1.207562236s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:18:39 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.207593745s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.......X.............X....X..........X..............] duration: 1.281364334s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:18:30 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.28139267s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 669.996114ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:18:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 670.021829ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X...X...X...................X......X.........] duration: 1.216076677s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:18:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.216098655s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.......X.............X....X..........X.............] duration: 964.034655ms trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:17:46 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 964.081249ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 664.056626ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:17:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 664.076035ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [.........................................X......X......XX.X.] duration: 2.577723576s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:17:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 2.577754845s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 5 out of 60 is failed (threshold=3) [..........................................X......X......XX.X] duration: 1m0.104080479s trace: https://jaeger.dev.storj.io/trace/1590f2704bce91c3 FAIL (2026-03-10 19:17:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== 24309 (104h16m10.405795345s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810180918 (104h16m10.406951948s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.810375072 (104h16m10.407146095s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.811053898 (104h16m10.407824931s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.812902940 (104h16m10.409673975s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.814775487 (104h16m10.411546518s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.816867635 (104h16m10.413638661s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.817857606 (104h16m10.414628634s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.824812389 (104h16m10.421583422s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.827480735 (104h16m10.424251767s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:12.836199462 (104h16m10.432970498s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Uploading piece... Mar 10 19:16:13.243336850 (104h16m10.840107865s) [upload=116703]: Next returned nil segment Mar 10 19:16:13.243351889 (104h16m10.840122913s) [upload=116703]: Waiting for error group managing segments... Mar 10 19:16:13.436915644 (104h16m11.033686667s) [noise=true , address=81.171.14.70:4039 , piece_id=YXUQXZHO2OT4BOXUN6PK43YOUWVCU6PHNQSJCZ2SMM4OSHP7K37Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.439011650 (104h16m11.035782674s) [noise=true , address=81.171.14.70:4026 , piece_id=EVTB3KKB4N5FEOF677JB4QFBBJ3MMDHISVXNNVSHTGTIZBDHRU7A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.441999794 (104h16m11.038770828s) [noise=true , address=81.171.14.70:4046 , piece_id=6W3G34RE3HSKDN4EOHUNQDY2F762XOACOT34TV25SPT2JLWWDJAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474619244 (104h16m11.071390271s) [noise=true , address=185.167.204.206:4040 , piece_id=ZMYU3SKVO3TFV2AZ6KPCAIGNK2XI6IJ2FZTLA3JNOEK6WFZSIGJQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.474766092 (104h16m11.071537115s) [noise=true , address=185.167.204.205:4030 , piece_id=ZJIIWH7Q5ZWIQOUX4DTI3G6JGM6KJQFVHROFNSGZE7ZZEQFX5EHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.479069763 (104h16m11.07584079s) [noise=true , address=185.167.204.206:4056 , piece_id=CLXTHIPM4FILPLM53PB2LBXBAXDH2EXVAZZN7F5SESIGLDTNGO3A, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.480030220 (104h16m11.076801248s) [noise=true , address=81.171.14.70:4047 , piece_id=TQXWY5ZSKL6GJLUHLIOBXDJFVEUS5S3QITHPY626JVKWZEGURJKA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.482792814 (104h16m11.079563838s) [noise=true , address=185.167.204.205:4008 , piece_id=JNDEB7NS4KJJZWYCHBYI3RCUCM4DBENDQFBTVNBZOHEKSMEMHMCA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497522407 (104h16m11.09429344s) [noise=true , address=89.149.201.147:4011 , piece_id=2R2DUIHFOHPZDNYWSRGZWKARTANWOERLSGWMKNJJ6OLJA7L2QL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.497964755 (104h16m11.094735779s) [noise=true , address=185.167.204.205:4007 , piece_id=BW2YCA4QBSPH43IINQXYS3ZO5SEDQ5LLHLZZYTIYZCM4N7ZFQUCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.501101180 (104h16m11.097872203s) [noise=true , address=185.167.204.205:4010 , piece_id=YGSFD6WSSIVIQ4QNNSRWXXQVZEIDKGGPOZN2C75DJO64ZSN6ZMUQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.503493705 (104h16m11.10026473s) [noise=true , address=185.167.204.206:4058 , piece_id=UPC4NU6E7SEBGRI7LIZNGJO4T5J7PESJSQQEJRCPJ4SZUFBF6IAQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.515561442 (104h16m11.112332469s) [noise=true , address=185.167.204.206:4060 , piece_id=VT2DC2HRTYXMJNJAYI6UM6CXW4LPOFZCJQN7ESU76RZ2G7BVSK5Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.519160405 (104h16m11.115931438s) [noise=true , address=185.167.204.205:4002 , piece_id=SWT45DFR3H54U23MDYAE5OFCYF54DCU57H3NVCZZR3RSZJLSQZLQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.540804171 (104h16m11.137575201s) [noise=true , address=185.167.204.205:4005 , piece_id=OJTSPVF27LLT3T2JF6XIE7I4ZY44HV2MAW4YO4MCGGD2AB3B2ZNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.551798501 (104h16m11.148569524s) [noise=true , address=185.167.204.205:4023 , piece_id=WZ46O2UBITJT5G7P7T4RAYZFJV6WOHA7R7TBKBOFAU5K2VQUY2YA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567089602 (104h16m11.163860628s) [noise=true , address=185.167.204.206:4065 , piece_id=XJOGBPFQZMPIOHQMULIT35PGOMZPAGIM6DXNT3ERHCQHBFY7GL2Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.567440890 (104h16m11.16421192s) [noise=true , address=185.167.204.205:4027 , piece_id=B4LXLWA7R5S3XVMK7GJCWBGMC6J6JR7P6VKE2IQJF4IPQZLLNSOA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.570406605 (104h16m11.16717763s) [noise=true , address=185.167.204.205:4003 , piece_id=63MW36UPXUTQY6F7ELHJ5ROMUYTYLX3EV772Y2IXXY574XIDMWHQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.596875470 (104h16m11.193646504s) [noise=true , address=185.167.204.205:4035 , piece_id=KWH36XETYIGHK5LZZRAHKWWJXQKC76U36SAWMAEN4B3MWYUIYRQQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.927555363 (104h16m11.524326395s) [noise=true , address=81.171.14.70:4041 , piece_id=4LDPRCUC3QSIM3F5LSGY55SWCWNIU7HKNEKCBYREJM5AOPNFX5TQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:13.999804802 (104h16m11.596575827s) [noise=true , address=89.149.201.147:4017 , piece_id=333QT34ZMXJ2CHFOW6ZFUV6NQ7HORKBQNTSE3MZ4OX2GRASE3PPQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:15.069819348 (104h16m12.666590387s) [noise=true , address=185.167.204.205:4009 , piece_id=P4VW3V6WA4XYVKCRV2YSROL6VAL5XSASEDZBFN44466YKDFTDOMQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:16:21.098353279 (104h16m18.695124303s) [noise=true , address=185.167.204.206:4063 , piece_id=W5RA3WHMUGGYWKSLOHDLNOFNGBABUDKVVTVUJJTGLC5BWUTIBRCQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: <nil> Mar 10 19:17:12.435335426 (104h17m10.032106453s) [noise=true , address=89.149.201.147:4009 , piece_id=HUMALSZKT2MZLJNVTBYLICHOWTD336XDAXOIMO3BXU3AOY2BIZDA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: read tcp 159.223.143.39:54332->89.149.201.147:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435433273 (104h17m10.032204303s) [noise=true , address=185.167.204.206:4067 , piece_id=NOPTVRD4PW3CROP5LYHCJZPHGYCGLETKCRWIYOKPE25FFEMGWPNQ, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:12khRPUvAHZgwi8rajx8TYpzujWybn3tcwZeG81R7JzHFRGL8PS, address:185.167.204.206:4067): protocol: expected piece hash; context deadline exceeded; EOF Mar 10 19:17:12.435547994 (104h17m10.032319018s) [noise=true , address=89.149.201.147:4013 , piece_id=ZZOWOTDSR7KWC7C36ZGQC7GU447PIBW2YRM6DA5QOAYUVXSE2WNA, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:125ZKP9g2cY7B3b8eyfHqoChUG2nYtz3nbFRGA6szgtACzT4M6J, address:89.149.201.147:4013): protocol: read tcp 159.223.143.39:49244->89.149.201.147:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435690156 (104h17m10.032461179s) [noise=true , address=89.149.201.147:4021 , piece_id=CWMJTXKTLB3UFDDA7T4SMUJ2LDO6XB2F7I62RJHCFCK43LZYMY6Q, seg_pos={0 0}, upload=116703]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:47460->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:17:12.435770391 (104h17m10.032541415s) [seg_pos={0 0}, upload=116703]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Mar 10 19:17:12.536875834 (104h17m10.133646857s) [upload=116703]: Done uploading segments... Mar 10 19:17:12.536913203 (104h17m10.133684228s) [upload=116703]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 1m0.104107653s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 688.677895ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:17:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 688.73288ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X...X...X...................X......X........] duration: 971.394037ms trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:17:00 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 971.426753ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.249060039s trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:16:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.24942314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.......X.............X....X..........X............] duration: 965.484473ms trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:16:40 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 965.504129ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X...X...X...................X......X.......] duration: 1.196470192s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:16:08 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.196520749s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 973.982257ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:16:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 974.014299ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X...X...X...................X......X......] duration: 1.196655373s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:15:40 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.19668926s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.......X.............X....X..........X...........] duration: 1.228156605s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:15:39 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.228185131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X......X......XX.] duration: 2.620178843s trace: https://jaeger.dev.storj.io/trace/412c6e599769150d FAIL (2026-03-10 19:15:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== ng piece... Mar 10 19:14:28.824376269 (104h14m26.421147302s) [noise=true , address=89.149.201.147:4021 , piece_id=MQMOET7ZCXVSTSIOPP6BJXY3ZPC6MDYRHBKIU2NUNUDYV5XOCUKA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.825312265 (104h14m26.422083302s) [noise=true , address=185.167.204.206:4065 , piece_id=G7VN6FKTO5LCSLZF5ZKSJSFTZWP6C7RM2J6UZMMESFZCQFD25KIQ, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.826224054 (104h14m26.422995087s) [noise=true , address=81.171.14.70:4037 , piece_id=UPFC6ATMDKTW54ILHGGR2RY4EZGQ4CCWJ5ERNAHXYJDGAUKHBOGA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.827209741 (104h14m26.423980768s) [noise=true , address=89.149.201.147:4007 , piece_id=4YSMLQGR3UHDX4WLM54QG3BKAKCGJRTO6437YLESNWZKGZSH743A, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.830106344 (104h14m26.426877372s) [noise=true , address=185.167.204.205:4030 , piece_id=X4IBA66LEKZUQ4ZHIVPEFSL655G372ZQUPKPQP2P2M2QC3KCBTBQ, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.833112659 (104h14m26.429883691s) [noise=true , address=185.167.204.206:4071 , piece_id=73DBZ25HZQEVMGG753HG4L4FMPYFK3I4E7ECIGSCEE7LFBVZEXIA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.835760685 (104h14m26.432531716s) [noise=true , address=89.149.201.147:4008 , piece_id=4BGX6FX7X65AXK2THEQBDB3D52NXWWZL5T4BFDBK5GEA3VJWIZUA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.841959368 (104h14m26.4387304s) [noise=true , address=185.167.204.205:4016 , piece_id=3QC3XU3MM4YFQ2A2CYXJQ4BAH323T6LK7MEIUOO3DFBGBRM5AZQA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.845146196 (104h14m26.44191723s) [noise=true , address=89.149.201.147:4009 , piece_id=IS54Y5BE3G5DBPBOIBNR7V5K3QXT3NMZ2OXYPEBV46NUIMZOH2QA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.847725276 (104h14m26.444496309s) [noise=true , address=185.167.204.206:4059 , piece_id=KRHVHDOVYYX5J57ML6GJNKSB76PTOBUAINXHU4BO4X3JGBVTLZIA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.850247449 (104h14m26.447018482s) [noise=true , address=89.149.201.147:4012 , piece_id=XGL6J5MOKHTFLHVJW2VBN3WN6VFLRPPEFXJSSWDEUBTJ452IFE4A, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:29.251249806 (104h14m26.848020829s) [upload=116672]: Next returned nil segment Mar 10 19:14:29.251264543 (104h14m26.848035569s) [upload=116672]: Waiting for error group managing segments... Mar 10 19:14:29.435752875 (104h14m27.032523888s) [noise=true , address=89.149.201.147:4019 , piece_id=ZJD2HN6ZDYXZE6HEAFQOJY7XKMO2TT4Y76VWU6DKMSM7A5OY6DZQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.441303744 (104h14m27.03807477s) [noise=true , address=185.167.204.206:4064 , piece_id=WEKNDS5Y4F725LDN6RE4RPJZSPMWO2FSV5BFA5ROPXW2Q7Z5GHHQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.463119569 (104h14m27.059890596s) [noise=true , address=81.171.14.70:4041 , piece_id=2CNWHPDSDYBJ3DRLZGXNX4V2GVQ56KWW3BEK2Y2U3YXB6UYEMFZQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.465829191 (104h14m27.062600205s) [noise=true , address=89.149.201.147:4016 , piece_id=76UC3L2FB5UY5HWGS4TNC3FSSSDF6NDO5SVQL6BUY4XTSBXENAZQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.465983078 (104h14m27.062754119s) [noise=true , address=81.171.14.70:4037 , piece_id=UPFC6ATMDKTW54ILHGGR2RY4EZGQ4CCWJ5ERNAHXYJDGAUKHBOGA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.474978589 (104h14m27.071749599s) [noise=true , address=81.171.14.70:4048 , piece_id=QYHGAH3CZMMUTHWC3NLHDDYHMKOE5B4UA6FEYFRBRIQZ4V3DO5CA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.476272905 (104h14m27.073043933s) [noise=true , address=89.149.201.147:4011 , piece_id=S66QK6QPID7Q3MZDRFEF2AE4Z2P3I25KK7IGYDXB3PO7AW3QM4AQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.480986870 (104h14m27.077757884s) [noise=true , address=185.167.204.206:4066 , piece_id=RRZFKTDHKFRUOS25ZRK5MU2FMCPX3XHYX3AJPWA5XA4AUHRVVOWA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.493355674 (104h14m27.09012671s) [noise=true , address=185.167.204.206:4068 , piece_id=DNCLXQLNBYPHU52K63TLPREUTNE6SGNGY6DOKESPWPGOGPVIQREA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.498475505 (104h14m27.095246529s) [noise=true , address=185.167.204.206:4071 , piece_id=73DBZ25HZQEVMGG753HG4L4FMPYFK3I4E7ECIGSCEE7LFBVZEXIA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.501976308 (104h14m27.098747431s) [noise=true , address=185.167.204.206:4065 , piece_id=G7VN6FKTO5LCSLZF5ZKSJSFTZWP6C7RM2J6UZMMESFZCQFD25KIQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.505460015 (104h14m27.102231044s) [noise=true , address=89.149.201.147:4009 , piece_id=IS54Y5BE3G5DBPBOIBNR7V5K3QXT3NMZ2OXYPEBV46NUIMZOH2QA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.522666148 (104h14m27.119437183s) [noise=true , address=89.149.201.147:4012 , piece_id=XGL6J5MOKHTFLHVJW2VBN3WN6VFLRPPEFXJSSWDEUBTJ452IFE4A, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.529043949 (104h14m27.125814982s) [noise=true , address=185.167.204.206:4070 , piece_id=5Y7H2QDZZ54B62NOWVC6TZKJU7SECBB7IUP565UF5TLHFEUIT4DQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.534466955 (104h14m27.131237981s) [noise=true , address=89.149.201.147:4006 , piece_id=L6IUS3Z2AJPXS3SN3MZSCLTBR474EICFTUALOFDHHL36IL4CRJKQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.534739105 (104h14m27.131510118s) [noise=true , address=89.149.201.147:4007 , piece_id=4YSMLQGR3UHDX4WLM54QG3BKAKCGJRTO6437YLESNWZKGZSH743A, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.537783700 (104h14m27.134554725s) [noise=true , address=185.167.204.205:4016 , piece_id=3QC3XU3MM4YFQ2A2CYXJQ4BAH323T6LK7MEIUOO3DFBGBRM5AZQA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.546670645 (104h14m27.143441681s) [noise=true , address=185.167.204.205:4030 , piece_id=X4IBA66LEKZUQ4ZHIVPEFSL655G372ZQUPKPQP2P2M2QC3KCBTBQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.553490799 (104h14m27.150261816s) [noise=true , address=185.167.204.205:4020 , piece_id=F4HF43S3ITNWQJK5QFLIKW27WU73N3RKJQRHTXUXKKT2INENPPMQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.920241123 (104h14m27.517012138s) [noise=true , address=89.149.201.147:4010 , piece_id=PRDTOLWZQTLTMISNRAHB5VBGEZJGHN2VG6WXU6R4YGIWAH4XYDKA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.936263271 (104h14m27.533034295s) [noise=true , address=81.171.14.70:4030 , piece_id=556GAGIB2CAQLPJWXMGPH3LHPRWKHPIKWOTCTMKVOW2FTYZX7UPQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:30.042008516 (104h14m27.638779529s) [noise=true , address=185.167.204.205:4029 , piece_id=FAWGJHM53FVWAECBEKUFMGHU6PXP6OOPOVDTB4VOBJDN5D6AHFQA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:30.348417633 (104h14m27.945188661s) [noise=true , address=185.167.204.205:4027 , piece_id=65XZEUIAISH4FWJFETAKQOX3BZSYBXZJDMETO3EX7H46B677XELQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:31.343140846 (104h14m28.93991186s) [noise=true , address=185.167.204.206:4072 , piece_id=63Z4GPJRMO7DCQ5SHPQ7ZQFU7TMYIPSYV7BHROAR3FDWETZHVI2A, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:34.752518163 (104h14m32.349289193s) [noise=true , address=185.167.204.206:4059 , piece_id=KRHVHDOVYYX5J57ML6GJNKSB76PTOBUAINXHU4BO4X3JGBVTLZIA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:15:28.443803999 (104h15m26.040575033s) [noise=true , address=89.149.201.147:4017 , piece_id=7HXI5MZLJW3L6BB4PLWNFAVDUCU7WDGKVIIN36WZVPBJ6ILQKJWA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:32888->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:15:28.443856408 (104h15m26.04062744s) [noise=true , address=89.149.201.147:4021 , piece_id=MQMOET7ZCXVSTSIOPP6BJXY3ZPC6MDYRHBKIU2NUNUDYV5XOCUKA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: write tcp 159.223.143.39:55774->89.149.201.147:4021: use of closed network connection; write tcp 159.223.143.39:55774->89.149.201.147:4021: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55774->89.149.201.147:4021: use of closed network connection Mar 10 19:15:28.444122745 (104h15m26.040893775s) [noise=true , address=89.149.201.147:4008 , piece_id=4BGX6FX7X65AXK2THEQBDB3D52NXWWZL5T4BFDBK5GEA3VJWIZUA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: ecclient: upload failed (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn, address:89.149.201.147:4008): protocol: read tcp 159.223.143.39:60956->89.149.201.147:4008: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:15:28.444155013 (104h15m26.040926029s) [seg_pos={0 0}, upload=116672]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:15:28.561525634 (104h15m26.158296667s) [upload=116672]: Done uploading segments... Mar 10 19:15:28.561573481 (104h15m26.158344513s) [upload=116672]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 2.620198936s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 4 out of 60 is failed (threshold=3) [............................................X......X......XX] duration: 1m0.121116454s trace: https://jaeger.dev.storj.io/trace/412c6e599769150d FAIL (2026-03-10 19:15:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== ng piece... Mar 10 19:14:28.824376269 (104h14m26.421147302s) [noise=true , address=89.149.201.147:4021 , piece_id=MQMOET7ZCXVSTSIOPP6BJXY3ZPC6MDYRHBKIU2NUNUDYV5XOCUKA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.825312265 (104h14m26.422083302s) [noise=true , address=185.167.204.206:4065 , piece_id=G7VN6FKTO5LCSLZF5ZKSJSFTZWP6C7RM2J6UZMMESFZCQFD25KIQ, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.826224054 (104h14m26.422995087s) [noise=true , address=81.171.14.70:4037 , piece_id=UPFC6ATMDKTW54ILHGGR2RY4EZGQ4CCWJ5ERNAHXYJDGAUKHBOGA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.827209741 (104h14m26.423980768s) [noise=true , address=89.149.201.147:4007 , piece_id=4YSMLQGR3UHDX4WLM54QG3BKAKCGJRTO6437YLESNWZKGZSH743A, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.830106344 (104h14m26.426877372s) [noise=true , address=185.167.204.205:4030 , piece_id=X4IBA66LEKZUQ4ZHIVPEFSL655G372ZQUPKPQP2P2M2QC3KCBTBQ, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.833112659 (104h14m26.429883691s) [noise=true , address=185.167.204.206:4071 , piece_id=73DBZ25HZQEVMGG753HG4L4FMPYFK3I4E7ECIGSCEE7LFBVZEXIA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.835760685 (104h14m26.432531716s) [noise=true , address=89.149.201.147:4008 , piece_id=4BGX6FX7X65AXK2THEQBDB3D52NXWWZL5T4BFDBK5GEA3VJWIZUA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.841959368 (104h14m26.4387304s) [noise=true , address=185.167.204.205:4016 , piece_id=3QC3XU3MM4YFQ2A2CYXJQ4BAH323T6LK7MEIUOO3DFBGBRM5AZQA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.845146196 (104h14m26.44191723s) [noise=true , address=89.149.201.147:4009 , piece_id=IS54Y5BE3G5DBPBOIBNR7V5K3QXT3NMZ2OXYPEBV46NUIMZOH2QA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.847725276 (104h14m26.444496309s) [noise=true , address=185.167.204.206:4059 , piece_id=KRHVHDOVYYX5J57ML6GJNKSB76PTOBUAINXHU4BO4X3JGBVTLZIA, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:28.850247449 (104h14m26.447018482s) [noise=true , address=89.149.201.147:4012 , piece_id=XGL6J5MOKHTFLHVJW2VBN3WN6VFLRPPEFXJSSWDEUBTJ452IFE4A, seg_pos={0 0}, upload=116672]: Uploading piece... Mar 10 19:14:29.251249806 (104h14m26.848020829s) [upload=116672]: Next returned nil segment Mar 10 19:14:29.251264543 (104h14m26.848035569s) [upload=116672]: Waiting for error group managing segments... Mar 10 19:14:29.435752875 (104h14m27.032523888s) [noise=true , address=89.149.201.147:4019 , piece_id=ZJD2HN6ZDYXZE6HEAFQOJY7XKMO2TT4Y76VWU6DKMSM7A5OY6DZQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.441303744 (104h14m27.03807477s) [noise=true , address=185.167.204.206:4064 , piece_id=WEKNDS5Y4F725LDN6RE4RPJZSPMWO2FSV5BFA5ROPXW2Q7Z5GHHQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.463119569 (104h14m27.059890596s) [noise=true , address=81.171.14.70:4041 , piece_id=2CNWHPDSDYBJ3DRLZGXNX4V2GVQ56KWW3BEK2Y2U3YXB6UYEMFZQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.465829191 (104h14m27.062600205s) [noise=true , address=89.149.201.147:4016 , piece_id=76UC3L2FB5UY5HWGS4TNC3FSSSDF6NDO5SVQL6BUY4XTSBXENAZQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.465983078 (104h14m27.062754119s) [noise=true , address=81.171.14.70:4037 , piece_id=UPFC6ATMDKTW54ILHGGR2RY4EZGQ4CCWJ5ERNAHXYJDGAUKHBOGA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.474978589 (104h14m27.071749599s) [noise=true , address=81.171.14.70:4048 , piece_id=QYHGAH3CZMMUTHWC3NLHDDYHMKOE5B4UA6FEYFRBRIQZ4V3DO5CA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.476272905 (104h14m27.073043933s) [noise=true , address=89.149.201.147:4011 , piece_id=S66QK6QPID7Q3MZDRFEF2AE4Z2P3I25KK7IGYDXB3PO7AW3QM4AQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.480986870 (104h14m27.077757884s) [noise=true , address=185.167.204.206:4066 , piece_id=RRZFKTDHKFRUOS25ZRK5MU2FMCPX3XHYX3AJPWA5XA4AUHRVVOWA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.493355674 (104h14m27.09012671s) [noise=true , address=185.167.204.206:4068 , piece_id=DNCLXQLNBYPHU52K63TLPREUTNE6SGNGY6DOKESPWPGOGPVIQREA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.498475505 (104h14m27.095246529s) [noise=true , address=185.167.204.206:4071 , piece_id=73DBZ25HZQEVMGG753HG4L4FMPYFK3I4E7ECIGSCEE7LFBVZEXIA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.501976308 (104h14m27.098747431s) [noise=true , address=185.167.204.206:4065 , piece_id=G7VN6FKTO5LCSLZF5ZKSJSFTZWP6C7RM2J6UZMMESFZCQFD25KIQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.505460015 (104h14m27.102231044s) [noise=true , address=89.149.201.147:4009 , piece_id=IS54Y5BE3G5DBPBOIBNR7V5K3QXT3NMZ2OXYPEBV46NUIMZOH2QA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.522666148 (104h14m27.119437183s) [noise=true , address=89.149.201.147:4012 , piece_id=XGL6J5MOKHTFLHVJW2VBN3WN6VFLRPPEFXJSSWDEUBTJ452IFE4A, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.529043949 (104h14m27.125814982s) [noise=true , address=185.167.204.206:4070 , piece_id=5Y7H2QDZZ54B62NOWVC6TZKJU7SECBB7IUP565UF5TLHFEUIT4DQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.534466955 (104h14m27.131237981s) [noise=true , address=89.149.201.147:4006 , piece_id=L6IUS3Z2AJPXS3SN3MZSCLTBR474EICFTUALOFDHHL36IL4CRJKQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.534739105 (104h14m27.131510118s) [noise=true , address=89.149.201.147:4007 , piece_id=4YSMLQGR3UHDX4WLM54QG3BKAKCGJRTO6437YLESNWZKGZSH743A, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.537783700 (104h14m27.134554725s) [noise=true , address=185.167.204.205:4016 , piece_id=3QC3XU3MM4YFQ2A2CYXJQ4BAH323T6LK7MEIUOO3DFBGBRM5AZQA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.546670645 (104h14m27.143441681s) [noise=true , address=185.167.204.205:4030 , piece_id=X4IBA66LEKZUQ4ZHIVPEFSL655G372ZQUPKPQP2P2M2QC3KCBTBQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.553490799 (104h14m27.150261816s) [noise=true , address=185.167.204.205:4020 , piece_id=F4HF43S3ITNWQJK5QFLIKW27WU73N3RKJQRHTXUXKKT2INENPPMQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.920241123 (104h14m27.517012138s) [noise=true , address=89.149.201.147:4010 , piece_id=PRDTOLWZQTLTMISNRAHB5VBGEZJGHN2VG6WXU6R4YGIWAH4XYDKA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:29.936263271 (104h14m27.533034295s) [noise=true , address=81.171.14.70:4030 , piece_id=556GAGIB2CAQLPJWXMGPH3LHPRWKHPIKWOTCTMKVOW2FTYZX7UPQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:30.042008516 (104h14m27.638779529s) [noise=true , address=185.167.204.205:4029 , piece_id=FAWGJHM53FVWAECBEKUFMGHU6PXP6OOPOVDTB4VOBJDN5D6AHFQA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:30.348417633 (104h14m27.945188661s) [noise=true , address=185.167.204.205:4027 , piece_id=65XZEUIAISH4FWJFETAKQOX3BZSYBXZJDMETO3EX7H46B677XELQ, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:31.343140846 (104h14m28.93991186s) [noise=true , address=185.167.204.206:4072 , piece_id=63Z4GPJRMO7DCQ5SHPQ7ZQFU7TMYIPSYV7BHROAR3FDWETZHVI2A, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:14:34.752518163 (104h14m32.349289193s) [noise=true , address=185.167.204.206:4059 , piece_id=KRHVHDOVYYX5J57ML6GJNKSB76PTOBUAINXHU4BO4X3JGBVTLZIA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: <nil> Mar 10 19:15:28.443803999 (104h15m26.040575033s) [noise=true , address=89.149.201.147:4017 , piece_id=7HXI5MZLJW3L6BB4PLWNFAVDUCU7WDGKVIIN36WZVPBJ6ILQKJWA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:32888->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:15:28.443856408 (104h15m26.04062744s) [noise=true , address=89.149.201.147:4021 , piece_id=MQMOET7ZCXVSTSIOPP6BJXY3ZPC6MDYRHBKIU2NUNUDYV5XOCUKA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: write tcp 159.223.143.39:55774->89.149.201.147:4021: use of closed network connection; write tcp 159.223.143.39:55774->89.149.201.147:4021: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55774->89.149.201.147:4021: use of closed network connection Mar 10 19:15:28.444122745 (104h15m26.040893775s) [noise=true , address=89.149.201.147:4008 , piece_id=4BGX6FX7X65AXK2THEQBDB3D52NXWWZL5T4BFDBK5GEA3VJWIZUA, seg_pos={0 0}, upload=116672]: Done uploading piece. err: ecclient: upload failed (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn, address:89.149.201.147:4008): protocol: read tcp 159.223.143.39:60956->89.149.201.147:4008: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:15:28.444155013 (104h15m26.040926029s) [seg_pos={0 0}, upload=116672]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:15:28.561525634 (104h15m26.158296667s) [upload=116672]: Done uploading segments... Mar 10 19:15:28.561573481 (104h15m26.158344513s) [upload=116672]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 1m0.121133869s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X...X...X...................X......X.....] duration: 912.370737ms trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:15:16 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 912.399229ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.037704853s trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:15:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.037732553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.......X.............X....X..........X..........] duration: 1.230318482s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:14:55 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.230347722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X...X...X...................X......X....] duration: 32.200933293s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:14:42 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 32.200966677s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X..................X........................................] duration: 658.953341ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:14:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 659.059776ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X......X......X] duration: 1m0.107931033s trace: https://jaeger.dev.storj.io/trace/1a3d7e89897fbe80 FAIL (2026-03-10 19:14:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=185.167.204.206:4053 , piece_id=OXVKRLX7BKHSMDFR7O35GKV2NBUI2IULI4ODQDRLIFSZQEUXSP6A, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:28.639165996 (104h13m26.235937029s) [noise=true , address=89.149.201.147:4010 , piece_id=OCHY66JN24WNWVV2I2HJ33AXRV7VQY753TMTUYI4SAQ2VFERAPMA, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:28.640935118 (104h13m26.237706142s) [noise=true , address=185.167.204.206:4054 , piece_id=RFMITHRAHUVCGVRSXP6SECCCB2Y2F4N4HBKWM3ZCZ6ML5USNHI4Q, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:28.642919830 (104h13m26.239690859s) [noise=true , address=81.171.14.70:4028 , piece_id=ZWZRXAV74RNW3AS3Z7BRCJEOR7QYRMXSCKMBIAPXUK2S2UU2U5OA, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:28.643232852 (104h13m26.240003885s) [noise=true , address=185.167.204.205:4019 , piece_id=UBQGZEAW7JEFZLOSSCRXM5S7YEWEBABWKFKFA3CQYX6GYNJRWKXQ, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:28.644567965 (104h13m26.241338988s) [noise=true , address=185.167.204.206:4066 , piece_id=YZJODNV3DGSEW27OTWOJAMBYQGNOISGMK5X6S7MSTWZYMF3XDSKQ, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:28.645296161 (104h13m26.242067185s) [noise=true , address=89.149.201.147:4023 , piece_id=PJ7BRSXWLZUFBCF3XYC34O7NMP43YNQFAZIPRQWRNOH4BZQX46ZA, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:28.646483548 (104h13m26.243254577s) [noise=true , address=89.149.201.147:4012 , piece_id=MLOCGSSTOY7J6A7DZUDURMI4D7UUMM2KRP5KBPHVTH6JVVT2XIKQ, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:28.647084194 (104h13m26.243855226s) [noise=true , address=81.171.14.70:4031 , piece_id=57VYCT6EZGAN5LMVTH3ZAR2LYJUE3PXNQE3WT4Z4MIX5N5TXVWMA, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:28.647182603 (104h13m26.243953637s) [noise=true , address=89.149.201.147:4003 , piece_id=6XKRTVBOUDITRT3KVDUNBNZR7QSNN2WI6PDEN6NMOOAH7QHGNNGA, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:28.655830806 (104h13m26.252601828s) [noise=true , address=81.171.14.70:4033 , piece_id=WQXKBHF6TCQKX4URCF6GDNBYEP7KQDMP4PDD5ISX3YP6E6VNNA4Q, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:28.658122860 (104h13m26.254893883s) [noise=true , address=185.167.204.206:4048 , piece_id=B4AAA35AWQ2YCQF4YX6IQTXJ3N47FXNEJTUMR2WV2NDCRYVLV7SQ, seg_pos={0 0}, upload=116654]: Uploading piece... Mar 10 19:13:29.101604505 (104h13m26.698375538s) [upload=116654]: Next returned nil segment Mar 10 19:13:29.101619617 (104h13m26.698390644s) [upload=116654]: Waiting for error group managing segments... Mar 10 19:13:29.271929126 (104h13m26.868700142s) [noise=true , address=81.171.14.70:4045 , piece_id=XTXOIQP2F7PFRWTOZOQ37SJYCH27DO2Q44INVKZYFRS5XBUPFLAA, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.277937915 (104h13m26.874708937s) [noise=true , address=81.171.14.70:4033 , piece_id=WQXKBHF6TCQKX4URCF6GDNBYEP7KQDMP4PDD5ISX3YP6E6VNNA4Q, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.279230867 (104h13m26.87600189s) [noise=true , address=185.167.204.206:4058 , piece_id=4I4FUGXC3NV3KYMJA6YOECR46WHOCLEWBPHG5GWERTSDVSDSS76Q, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.281786038 (104h13m26.878557062s) [noise=true , address=185.167.204.206:4053 , piece_id=OXVKRLX7BKHSMDFR7O35GKV2NBUI2IULI4ODQDRLIFSZQEUXSP6A, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.310398626 (104h13m26.90716964s) [noise=true , address=185.167.204.206:4066 , piece_id=YZJODNV3DGSEW27OTWOJAMBYQGNOISGMK5X6S7MSTWZYMF3XDSKQ, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.310974375 (104h13m26.907745388s) [noise=true , address=185.167.204.205:4028 , piece_id=HQF6RFF5XBILZQ3G3OG53B4UTQELFHMVR6XAYUB47ALDKGOKWKKQ, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.311828850 (104h13m26.908599863s) [noise=true , address=89.149.201.147:4023 , piece_id=PJ7BRSXWLZUFBCF3XYC34O7NMP43YNQFAZIPRQWRNOH4BZQX46ZA, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.313335557 (104h13m26.910106571s) [noise=true , address=89.149.201.147:4010 , piece_id=OCHY66JN24WNWVV2I2HJ33AXRV7VQY753TMTUYI4SAQ2VFERAPMA, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.329992845 (104h13m26.926763876s) [noise=true , address=185.167.204.206:4038 , piece_id=CQCCNMM6FNEIE7CH4HRZNTFWOPI5MBOEMC372IQFHWCU3I4CQJPQ, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.356200635 (104h13m26.952971707s) [noise=true , address=185.167.204.206:4057 , piece_id=B47WROR23EVJYVVVF2BNYU2ZVHQMIMWXYE3DLCE5PDN6W3DOFMPQ, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.361842850 (104h13m26.958613876s) [noise=true , address=185.167.204.206:4054 , piece_id=RFMITHRAHUVCGVRSXP6SECCCB2Y2F4N4HBKWM3ZCZ6ML5USNHI4Q, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.380796646 (104h13m26.977567669s) [noise=true , address=185.167.204.205:4016 , piece_id=Z35Z7SIALOOVQ3V3XYUKB37AUQCOST7HLDXFC4G7HLBYOPIPGOJA, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.398364352 (104h13m26.995135375s) [noise=true , address=185.167.204.206:4048 , piece_id=B4AAA35AWQ2YCQF4YX6IQTXJ3N47FXNEJTUMR2WV2NDCRYVLV7SQ, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.405972442 (104h13m27.002743469s) [noise=true , address=185.167.204.205:4019 , piece_id=UBQGZEAW7JEFZLOSSCRXM5S7YEWEBABWKFKFA3CQYX6GYNJRWKXQ, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.418340813 (104h13m27.015111825s) [noise=true , address=185.167.204.206:4043 , piece_id=HZIAMQYLUUPMDZ5AGQSTEJPLBZA4YVJ45QXQQZBSWF6LMDYSVLGA, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.749547017 (104h13m27.34631804s) [noise=true , address=89.149.201.147:4003 , piece_id=6XKRTVBOUDITRT3KVDUNBNZR7QSNN2WI6PDEN6NMOOAH7QHGNNGA, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.754161876 (104h13m27.3509329s) [noise=true , address=81.171.14.70:4031 , piece_id=57VYCT6EZGAN5LMVTH3ZAR2LYJUE3PXNQE3WT4Z4MIX5N5TXVWMA, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.766146073 (104h13m27.362917106s) [noise=true , address=81.171.14.70:4027 , piece_id=5ZOPZQQNP6N3JJYULSYX4RSVFAIMPP464TIHFY7WAVMGBVHS7CKQ, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.774190489 (104h13m27.370961512s) [noise=true , address=89.149.201.147:4006 , piece_id=7AYRKNYX4ZOCVF3C2WACTXHJH24C5DZ4JE5ZDGJ4E4KUU74CGCOA, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.779466041 (104h13m27.376237064s) [noise=true , address=89.149.201.147:4024 , piece_id=4MBVCOROCPXBGD62TKRPXYCSDN2LTPYRIDTG34E2EDUGRT5Q5PYQ, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.785025012 (104h13m27.381796034s) [noise=true , address=89.149.201.147:4019 , piece_id=CTJLJAH4DFYUP7DYZKBFFZAO7UT2QP6EWEYEI36FOOQTMRUBT5FQ, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.799602268 (104h13m27.39637329s) [noise=true , address=81.171.14.70:4037 , piece_id=EMGBBOTKYAOPM43R5E6OJ7DQ7SGJAN57B7RW22AKGBZSZPZASJGQ, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.823554444 (104h13m27.420325474s) [noise=true , address=81.171.14.70:4028 , piece_id=ZWZRXAV74RNW3AS3Z7BRCJEOR7QYRMXSCKMBIAPXUK2S2UU2U5OA, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:29.861279421 (104h13m27.458050451s) [noise=true , address=89.149.201.147:4016 , piece_id=6XZJ7XEKFYY5KHMV3QGWKTM3GUZIB5S6DSHN4PUIDEZOMYDGEN4Q, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:13:32.832122191 (104h13m30.428893232s) [noise=true , address=185.167.204.205:4023 , piece_id=JAGOGUYGE5L66XU72DSBAY6MBUV2EEAUCZT2RRCPUORODKS6JR5Q, seg_pos={0 0}, upload=116654]: Done uploading piece. err: <nil> Mar 10 19:14:28.263703619 (104h14m25.860474646s) [noise=true , address=89.149.201.147:4012 , piece_id=MLOCGSSTOY7J6A7DZUDURMI4D7UUMM2KRP5KBPHVTH6JVVT2XIKQ, seg_pos={0 0}, upload=116654]: Done uploading piece. err: ecclient: upload failed (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv, address:89.149.201.147:4012): protocol: read tcp 159.223.143.39:53924->89.149.201.147:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:14:28.264464578 (104h14m25.861235611s) [noise=true , address=89.149.201.147:4017 , piece_id=NMOM2JQNSDALIMJ24UEW3AMT4OHUFI7DQZ2TRSSDISISWAVGWCUA, seg_pos={0 0}, upload=116654]: Done uploading piece. err: ecclient: upload failed (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ, address:89.149.201.147:4017): protocol: read tcp 159.223.143.39:55218->89.149.201.147:4017: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:14:28.265205056 (104h14m25.861976089s) [noise=true , address=89.149.201.147:4011 , piece_id=ZBOS3NWVB4WDROPB7K2C7QZSQLNIYZ7BRHD6EKYBK2W7SMU5BY2Q, seg_pos={0 0}, upload=116654]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: read tcp 159.223.143.39:55260->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Mar 10 19:14:28.265240990 (104h14m25.862012013s) [seg_pos={0 0}, upload=116654]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:14:28.365319741 (104h14m25.962090756s) [upload=116654]: Done uploading segments... Mar 10 19:14:28.365440642 (104h14m25.962211678s) [upload=116654]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 1m0.108051661s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.......X.............X....X..........X.........] duration: 1.291564905s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:13:45 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.291615358s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X..................X.......................................] duration: 722.488751ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:13:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 722.516986ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X...X...X...................X......X...] duration: 1.218898639s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:13:23 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.218930157s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X......X......] duration: 8.153030534s trace: https://jaeger.dev.storj.io/trace/60e038fc0a6bf5b6 FAIL (2026-03-10 19:12:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== pload=116536]: Uploading piece... Mar 10 19:07:08.052812464 (104h7m5.649583488s) [noise=true , address=89.149.201.147:4012 , piece_id=6FCDV7M3V6OYZNCRXFKU7AZFQOIOUDCTRXZFZRVKY5D653CVVSRQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053192041 (104h7m5.649963064s) [noise=true , address=185.167.204.206:4053 , piece_id=FL3WYO7QV7WVPNTC5L2HQIZLZ5Y5KXCCCHTFZ4WLMHLYY2DG3RXQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053534479 (104h7m5.650305505s) [noise=true , address=81.171.14.70:4030 , piece_id=CSNUZOIPMFMJWTOFAWBPXSSET7ITOX4XHSVP5OPQQAQOTFCHOJ4A, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053903216 (104h7m5.650674239s) [noise=true , address=185.167.204.206:4065 , piece_id=73HCX24X7GU4EXZHRDM7BV6UNKA5FNYVCNHWXIQXW7OV4VETWDNA, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.054577271 (104h7m5.651348304s) [noise=true , address=185.167.204.206:4049 , piece_id=NI52TPUMI4JVQ5TVMNQKN6IW3YZSU4EBXCLECA4PP7YSPVSNMTZQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.054916818 (104h7m5.651687833s) [noise=true , address=81.171.14.70:4039 , piece_id=B5KA4MB6AQTBQIGJBRHR2BVUOMRH5ISISW6GRTCCQ7KVDYGXWOYA, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055013708 (104h7m5.651784734s) [noise=true , address=185.167.204.206:4047 , piece_id=2LYHINYE53WQOF34OLI2KA4HUNEBU2SADVHQFMZBVZB4MPSAV6RQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055398681 (104h7m5.652169696s) [noise=true , address=89.149.201.147:4014 , piece_id=2CDEUE6S3DFXJ57KQIAO7ZUTBXBZLL7ORFR2SJ5PLGGC7WSXCS5Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055413527 (104h7m5.652184552s) [noise=true , address=81.171.14.70:4031 , piece_id=22JBHNL7OQRH4TOBPKYIFH5ZVIIL2LIJDFQGGLXBW6UGTDBC4EBQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055769117 (104h7m5.652540146s) [noise=true , address=81.171.14.70:4026 , piece_id=CVYTYXF7RPZS4FMUPFUMINXJBXY42NOI3QHI5A5YSU26FTSGA72Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055793969 (104h7m5.652564986s) [noise=true , address=89.149.201.147:4013 , piece_id=T2VCAXBO5MJW5THHW7AOSSZJ3FJUV3WCVMMHSAONG7VXY6OCSZ2Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.059615569 (104h7m5.656386589s) [noise=true , address=89.149.201.147:4007 , piece_id=ICSU7CLO5VDMAJSX47CCMF3ZCHPGEZT6TWV3PTD25WCWB4MCO7JQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.483435998 (104h7m6.080207019s) [upload=116536]: Next returned nil segment Mar 10 19:07:08.483463545 (104h7m6.080234571s) [upload=116536]: Waiting for error group managing segments... Mar 10 19:07:08.670085108 (104h7m6.266856132s) [noise=true , address=81.171.14.70:4030 , piece_id=CSNUZOIPMFMJWTOFAWBPXSSET7ITOX4XHSVP5OPQQAQOTFCHOJ4A, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.671751688 (104h7m6.26852271s) [noise=true , address=81.171.14.70:4039 , piece_id=B5KA4MB6AQTBQIGJBRHR2BVUOMRH5ISISW6GRTCCQ7KVDYGXWOYA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.675713790 (104h7m6.272484814s) [noise=true , address=81.171.14.70:4026 , piece_id=CVYTYXF7RPZS4FMUPFUMINXJBXY42NOI3QHI5A5YSU26FTSGA72Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.678729994 (104h7m6.275501017s) [noise=true , address=81.171.14.70:4031 , piece_id=22JBHNL7OQRH4TOBPKYIFH5ZVIIL2LIJDFQGGLXBW6UGTDBC4EBQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.679660985 (104h7m6.276432008s) [noise=true , address=185.167.204.206:4056 , piece_id=QNKSFJNUWFOKEEO4TBMWZQSMCABV3SES7XMQOBLIRB3Y4IEDNUZQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.704364612 (104h7m6.301135641s) [noise=true , address=89.149.201.147:4013 , piece_id=T2VCAXBO5MJW5THHW7AOSSZJ3FJUV3WCVMMHSAONG7VXY6OCSZ2Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.713873267 (104h7m6.310644294s) [noise=true , address=81.171.14.70:4036 , piece_id=EER6RSAHNMHEVN4XAX32OY3FUI4OK24I5LUFLPYHWDS34SNGPQRQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.716489526 (104h7m6.313260548s) [noise=true , address=185.167.204.205:4013 , piece_id=DLV3JPA46SEE2MLT2OUCYTTGRKE3YIFQ7J53SKCTKMZYDFRLQOVQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.720880819 (104h7m6.317651844s) [noise=true , address=185.167.204.206:4065 , piece_id=73HCX24X7GU4EXZHRDM7BV6UNKA5FNYVCNHWXIQXW7OV4VETWDNA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.721091989 (104h7m6.317863022s) [noise=true , address=81.171.14.70:4027 , piece_id=XUI3PS252PXEW6RU2KMS6JKB3VRWFAP4AH4MV2VHFNA2EY6BIG5A, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.722224062 (104h7m6.318995085s) [noise=true , address=89.149.201.147:4019 , piece_id=55XGKBTVO53PNYUPYO4AYAA7RQXNVCKHSB7VJ5FT474JJEOPIMCQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.726774177 (104h7m6.323545218s) [noise=true , address=89.149.201.147:4006 , piece_id=4275N735EX3ESSBGQMG6UURDYKZQT4KDTGI3UTZFPLZNUKMRBXEQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.740457541 (104h7m6.337228557s) [noise=true , address=185.167.204.206:4053 , piece_id=FL3WYO7QV7WVPNTC5L2HQIZLZ5Y5KXCCCHTFZ4WLMHLYY2DG3RXQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.764058724 (104h7m6.360829747s) [noise=true , address=185.167.204.206:4049 , piece_id=NI52TPUMI4JVQ5TVMNQKN6IW3YZSU4EBXCLECA4PP7YSPVSNMTZQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.764920220 (104h7m6.361691243s) [noise=true , address=185.167.204.206:4047 , piece_id=2LYHINYE53WQOF34OLI2KA4HUNEBU2SADVHQFMZBVZB4MPSAV6RQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.777059183 (104h7m6.373830232s) [noise=true , address=185.167.204.205:4005 , piece_id=WG55KC6CHZZTDQGQX4CKLT5IQN3PZS5HST6ADPV3K7IURYYMOPFQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.872009693 (104h7m6.468780719s) [noise=true , address=89.149.201.147:4012 , piece_id=6FCDV7M3V6OYZNCRXFKU7AZFQOIOUDCTRXZFZRVKY5D653CVVSRQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.184413047 (104h7m6.781184061s) [noise=true , address=81.171.14.70:4025 , piece_id=Q4ZIYTW5CSVBRXLFZFBUUZHBC4ER63OE7CNPZP5UBLNS2AFFT7HQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.218611147 (104h7m6.81538216s) [noise=true , address=81.171.14.70:4028 , piece_id=W2MHTUKXME6LLSBOCLUJWQAGEZM34ITWB2QVSO3NQWUFYIUSEY3Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.231788290 (104h7m6.828559319s) [noise=true , address=89.149.201.147:4024 , piece_id=SM35AGGVT24VDC46LEL5SR6XE3Y4WVEWFQJSFZS2K2XRMF64IWQA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.243351966 (104h7m6.840122993s) [noise=true , address=185.167.204.206:4045 , piece_id=KNNFTU6BBNDSTJPWTEKJIIYZQLRYL3T7SBSTFUUYI6EGPL2LFFBA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.911871668 (104h7m7.508642693s) [noise=true , address=185.167.204.205:4012 , piece_id=4G3LH5G6IGVEI3APRDKIYMEI7BT5H7MQYHXOVCLWAZZD433BV5YQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.957977058 (104h7m7.554748082s) [noise=true , address=185.167.204.206:4039 , piece_id=KC66HUYTZO4HBTMTPZF7OBPGEO2NVOHN6MXA256LQAOYM36AOSNQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:10.427584748 (104h7m8.024355769s) [noise=true , address=185.167.204.205:4001 , piece_id=WFICPLSRPVAWA2PTJGUP32TMVADLZ4M3E2YAOF64FJQFJBT2ATCQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:22.545770375 (104h7m20.142541405s) [noise=true , address=185.167.204.205:4019 , piece_id=7I4GCV5ACILSRHAUTT2DUPWORYIZAAMLGXQEDDSQWXWO66I3N7ZA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:08:07.676830562 (104h8m5.273601595s) [noise=true , address=89.149.201.147:4007 , piece_id=ICSU7CLO5VDMAJSX47CCMF3ZCHPGEZT6TWV3PTD25WCWB4MCO7JQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:38552->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.676856227 (104h8m5.273627257s) [noise=true , address=89.149.201.147:4021 , piece_id=2IYNAWS3HJWU7Z53DQ77BPWUXWA2W26Q5PTKUMLNXUGNWRK7NDLQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:35864->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.677137384 (104h8m5.273908417s) [noise=true , address=89.149.201.147:4014 , piece_id=2CDEUE6S3DFXJ57KQIAO7ZUTBXBZLL7ORFR2SJ5PLGGC7WSXCS5Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: read tcp 159.223.143.39:48614->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.677172053 (104h8m5.273943066s) [seg_pos={0 0}, upload=116536]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:08:07.777852887 (104h8m5.3746239s) [upload=116536]: Done uploading segments... Mar 10 19:08:07.777889836 (104h8m5.37466086s) [upload=116536]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 8.153057047s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.......X.............X....X..........X........] duration: 1.264233309s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:12:43 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.264261628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X..................X......................................] duration: 619.918269ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:12:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 619.967869ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.......X.............X....X..........X.......] duration: 1.253536084s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:12:14 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.253564684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X...X...X...................X......X..] duration: 1.272275922s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:12:10 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.272305849s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X......X.....] duration: 7.543477894s trace: https://jaeger.dev.storj.io/trace/60e038fc0a6bf5b6 FAIL (2026-03-10 19:11:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== pload=116536]: Uploading piece... Mar 10 19:07:08.052812464 (104h7m5.649583488s) [noise=true , address=89.149.201.147:4012 , piece_id=6FCDV7M3V6OYZNCRXFKU7AZFQOIOUDCTRXZFZRVKY5D653CVVSRQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053192041 (104h7m5.649963064s) [noise=true , address=185.167.204.206:4053 , piece_id=FL3WYO7QV7WVPNTC5L2HQIZLZ5Y5KXCCCHTFZ4WLMHLYY2DG3RXQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053534479 (104h7m5.650305505s) [noise=true , address=81.171.14.70:4030 , piece_id=CSNUZOIPMFMJWTOFAWBPXSSET7ITOX4XHSVP5OPQQAQOTFCHOJ4A, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053903216 (104h7m5.650674239s) [noise=true , address=185.167.204.206:4065 , piece_id=73HCX24X7GU4EXZHRDM7BV6UNKA5FNYVCNHWXIQXW7OV4VETWDNA, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.054577271 (104h7m5.651348304s) [noise=true , address=185.167.204.206:4049 , piece_id=NI52TPUMI4JVQ5TVMNQKN6IW3YZSU4EBXCLECA4PP7YSPVSNMTZQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.054916818 (104h7m5.651687833s) [noise=true , address=81.171.14.70:4039 , piece_id=B5KA4MB6AQTBQIGJBRHR2BVUOMRH5ISISW6GRTCCQ7KVDYGXWOYA, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055013708 (104h7m5.651784734s) [noise=true , address=185.167.204.206:4047 , piece_id=2LYHINYE53WQOF34OLI2KA4HUNEBU2SADVHQFMZBVZB4MPSAV6RQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055398681 (104h7m5.652169696s) [noise=true , address=89.149.201.147:4014 , piece_id=2CDEUE6S3DFXJ57KQIAO7ZUTBXBZLL7ORFR2SJ5PLGGC7WSXCS5Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055413527 (104h7m5.652184552s) [noise=true , address=81.171.14.70:4031 , piece_id=22JBHNL7OQRH4TOBPKYIFH5ZVIIL2LIJDFQGGLXBW6UGTDBC4EBQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055769117 (104h7m5.652540146s) [noise=true , address=81.171.14.70:4026 , piece_id=CVYTYXF7RPZS4FMUPFUMINXJBXY42NOI3QHI5A5YSU26FTSGA72Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055793969 (104h7m5.652564986s) [noise=true , address=89.149.201.147:4013 , piece_id=T2VCAXBO5MJW5THHW7AOSSZJ3FJUV3WCVMMHSAONG7VXY6OCSZ2Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.059615569 (104h7m5.656386589s) [noise=true , address=89.149.201.147:4007 , piece_id=ICSU7CLO5VDMAJSX47CCMF3ZCHPGEZT6TWV3PTD25WCWB4MCO7JQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.483435998 (104h7m6.080207019s) [upload=116536]: Next returned nil segment Mar 10 19:07:08.483463545 (104h7m6.080234571s) [upload=116536]: Waiting for error group managing segments... Mar 10 19:07:08.670085108 (104h7m6.266856132s) [noise=true , address=81.171.14.70:4030 , piece_id=CSNUZOIPMFMJWTOFAWBPXSSET7ITOX4XHSVP5OPQQAQOTFCHOJ4A, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.671751688 (104h7m6.26852271s) [noise=true , address=81.171.14.70:4039 , piece_id=B5KA4MB6AQTBQIGJBRHR2BVUOMRH5ISISW6GRTCCQ7KVDYGXWOYA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.675713790 (104h7m6.272484814s) [noise=true , address=81.171.14.70:4026 , piece_id=CVYTYXF7RPZS4FMUPFUMINXJBXY42NOI3QHI5A5YSU26FTSGA72Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.678729994 (104h7m6.275501017s) [noise=true , address=81.171.14.70:4031 , piece_id=22JBHNL7OQRH4TOBPKYIFH5ZVIIL2LIJDFQGGLXBW6UGTDBC4EBQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.679660985 (104h7m6.276432008s) [noise=true , address=185.167.204.206:4056 , piece_id=QNKSFJNUWFOKEEO4TBMWZQSMCABV3SES7XMQOBLIRB3Y4IEDNUZQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.704364612 (104h7m6.301135641s) [noise=true , address=89.149.201.147:4013 , piece_id=T2VCAXBO5MJW5THHW7AOSSZJ3FJUV3WCVMMHSAONG7VXY6OCSZ2Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.713873267 (104h7m6.310644294s) [noise=true , address=81.171.14.70:4036 , piece_id=EER6RSAHNMHEVN4XAX32OY3FUI4OK24I5LUFLPYHWDS34SNGPQRQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.716489526 (104h7m6.313260548s) [noise=true , address=185.167.204.205:4013 , piece_id=DLV3JPA46SEE2MLT2OUCYTTGRKE3YIFQ7J53SKCTKMZYDFRLQOVQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.720880819 (104h7m6.317651844s) [noise=true , address=185.167.204.206:4065 , piece_id=73HCX24X7GU4EXZHRDM7BV6UNKA5FNYVCNHWXIQXW7OV4VETWDNA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.721091989 (104h7m6.317863022s) [noise=true , address=81.171.14.70:4027 , piece_id=XUI3PS252PXEW6RU2KMS6JKB3VRWFAP4AH4MV2VHFNA2EY6BIG5A, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.722224062 (104h7m6.318995085s) [noise=true , address=89.149.201.147:4019 , piece_id=55XGKBTVO53PNYUPYO4AYAA7RQXNVCKHSB7VJ5FT474JJEOPIMCQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.726774177 (104h7m6.323545218s) [noise=true , address=89.149.201.147:4006 , piece_id=4275N735EX3ESSBGQMG6UURDYKZQT4KDTGI3UTZFPLZNUKMRBXEQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.740457541 (104h7m6.337228557s) [noise=true , address=185.167.204.206:4053 , piece_id=FL3WYO7QV7WVPNTC5L2HQIZLZ5Y5KXCCCHTFZ4WLMHLYY2DG3RXQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.764058724 (104h7m6.360829747s) [noise=true , address=185.167.204.206:4049 , piece_id=NI52TPUMI4JVQ5TVMNQKN6IW3YZSU4EBXCLECA4PP7YSPVSNMTZQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.764920220 (104h7m6.361691243s) [noise=true , address=185.167.204.206:4047 , piece_id=2LYHINYE53WQOF34OLI2KA4HUNEBU2SADVHQFMZBVZB4MPSAV6RQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.777059183 (104h7m6.373830232s) [noise=true , address=185.167.204.205:4005 , piece_id=WG55KC6CHZZTDQGQX4CKLT5IQN3PZS5HST6ADPV3K7IURYYMOPFQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.872009693 (104h7m6.468780719s) [noise=true , address=89.149.201.147:4012 , piece_id=6FCDV7M3V6OYZNCRXFKU7AZFQOIOUDCTRXZFZRVKY5D653CVVSRQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.184413047 (104h7m6.781184061s) [noise=true , address=81.171.14.70:4025 , piece_id=Q4ZIYTW5CSVBRXLFZFBUUZHBC4ER63OE7CNPZP5UBLNS2AFFT7HQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.218611147 (104h7m6.81538216s) [noise=true , address=81.171.14.70:4028 , piece_id=W2MHTUKXME6LLSBOCLUJWQAGEZM34ITWB2QVSO3NQWUFYIUSEY3Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.231788290 (104h7m6.828559319s) [noise=true , address=89.149.201.147:4024 , piece_id=SM35AGGVT24VDC46LEL5SR6XE3Y4WVEWFQJSFZS2K2XRMF64IWQA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.243351966 (104h7m6.840122993s) [noise=true , address=185.167.204.206:4045 , piece_id=KNNFTU6BBNDSTJPWTEKJIIYZQLRYL3T7SBSTFUUYI6EGPL2LFFBA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.911871668 (104h7m7.508642693s) [noise=true , address=185.167.204.205:4012 , piece_id=4G3LH5G6IGVEI3APRDKIYMEI7BT5H7MQYHXOVCLWAZZD433BV5YQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.957977058 (104h7m7.554748082s) [noise=true , address=185.167.204.206:4039 , piece_id=KC66HUYTZO4HBTMTPZF7OBPGEO2NVOHN6MXA256LQAOYM36AOSNQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:10.427584748 (104h7m8.024355769s) [noise=true , address=185.167.204.205:4001 , piece_id=WFICPLSRPVAWA2PTJGUP32TMVADLZ4M3E2YAOF64FJQFJBT2ATCQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:22.545770375 (104h7m20.142541405s) [noise=true , address=185.167.204.205:4019 , piece_id=7I4GCV5ACILSRHAUTT2DUPWORYIZAAMLGXQEDDSQWXWO66I3N7ZA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:08:07.676830562 (104h8m5.273601595s) [noise=true , address=89.149.201.147:4007 , piece_id=ICSU7CLO5VDMAJSX47CCMF3ZCHPGEZT6TWV3PTD25WCWB4MCO7JQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:38552->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.676856227 (104h8m5.273627257s) [noise=true , address=89.149.201.147:4021 , piece_id=2IYNAWS3HJWU7Z53DQ77BPWUXWA2W26Q5PTKUMLNXUGNWRK7NDLQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:35864->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.677137384 (104h8m5.273908417s) [noise=true , address=89.149.201.147:4014 , piece_id=2CDEUE6S3DFXJ57KQIAO7ZUTBXBZLL7ORFR2SJ5PLGGC7WSXCS5Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: read tcp 159.223.143.39:48614->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.677172053 (104h8m5.273943066s) [seg_pos={0 0}, upload=116536]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:08:07.777852887 (104h8m5.3746239s) [upload=116536]: Done uploading segments... Mar 10 19:08:07.777889836 (104h8m5.37466086s) [upload=116536]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 7.543500837s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X.......X.............X....X..........X......] duration: 1.259546215s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:11:32 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.259571702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X...X...X...................X......X.] duration: 1.274406557s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:11:28 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1.274426731s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................X...X...X...................X......X] duration: 1m0.031799901s trace: https://jaeger.dev.storj.io/trace/69062766ac2178dd FAIL (2026-03-10 19:11:26 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Mar 10 19:10:26.813115721 (104h10m24.409886745s) [upload=116599]: Starting upload Mar 10 19:10:26.813211715 (104h10m24.409982738s) [upload=116599]: Done starting upload Mar 10 19:10:26.813505664 (104h10m24.410276708s) [upload=116599]: Uploading segments... Mar 10 19:10:26.813650528 (104h10m24.410421539s) [upload=116599]: Got next segment. Inline: false Mar 10 19:10:26.813743195 (104h10m24.410514218s) [upload=116599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:10:26.813750388 (104h10m24.410521411s) [upload=116599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:10:26.846063672 (104h10m24.442834685s) [seg_pos={0 0}, upload=116599]: Begin upload segment... Mar 10 19:10:26.846148290 (104h10m24.442919315s) [seg_pos={0 0}, upload=116599]: Done begin upload segment. Mar 10 19:10:26.846228010 (104h10m24.442999069s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846301186 (104h10m24.44307221s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.846940319 (104h10m24.443711359s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.849741507 (104h10m24.446512533s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.850583955 (104h10m24.447354987s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851500001 (104h10m24.448271012s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.851893521 (104h10m24.448664555s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853333272 (104h10m24.450104308s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.853969417 (104h10m24.450740435s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854283058 (104h10m24.451054099s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.854903526 (104h10m24.45167454s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855071429 (104h10m24.451842462s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855704995 (104h10m24.452476028s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.855767508 (104h10m24.45253852s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.856577407 (104h10m24.453348417s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.857565751 (104h10m24.454336779s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.858400866 (104h10m24.455171879s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.859258131 (104h10m24.456029145s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Uploading piece... Mar 10 19:10:26.934798292 (104h10m24.531569311s) [upload=116599]: Next returned nil segment Mar 10 19:10:26.934817790 (104h10m24.531588824s) [upload=116599]: Waiting for error group managing segments... Mar 10 19:10:26.999719507 (104h10m24.59649054s) [noise=true , address=145.79.227.2:4133 , piece_id=AXG6E4SMSMKGVOINK4HMHTXL6GRKQWU6NJOP633KYOKQ3HQHMFOA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.030173609 (104h10m24.626944622s) [noise=true , address=5.231.73.4:4012 , piece_id=5RZOPQ6PFL42RJPI4ZR4GS756AU332JPC34SVWK4M2FELGI3O7CA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.238694294 (104h10m24.83546533s) [noise=true , address=167.17.72.70:4758 , piece_id=3OZLHHR5C6OX4EJK23YRLEGJOJ3IXWPR35CZ6MQARXGX5NNGXUGQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.294399709 (104h10m24.891170737s) [noise=true , address=167.17.72.68:4654 , piece_id=WHKUMMAPP7ZSKFMAGKFHINHWIICOCMREK75F3EON723TKLAWDDAQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.393989371 (104h10m24.990760384s) [noise=true , address=79.127.205.233:4154 , piece_id=LDMVPR6RMS2F3YNIMVH2ELREQMP6WMEXRW5UDBHTZIPZEPWRD4BQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.451000332 (104h10m25.047771355s) [noise=true , address=193.149.164.69:4092 , piece_id=T63NKUCMVMLADZYIMCW3FMJLYURBF7YVSMQ7AFLCYOU2K3C5RYNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.453000338 (104h10m25.04977136s) [noise=true , address=193.149.164.70:4116 , piece_id=FVAICHAZFMXEXK5CQ3SXYQ46T54PQBGYVPPBSKZ7WOHSGOKPHWRQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.472395366 (104h10m25.069166393s) [noise=true , address=193.149.164.71:4306 , piece_id=SJ5TPY5AZVF55AK3MRAT6K5ZPA4V6N2JXBYILVM3NHTCTRZI6MLQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.486423055 (104h10m25.083194088s) [noise=true , address=64.49.13.140:4507 , piece_id=6PNJPCU3KVM4ZFBTXQVZG26IEG57G3HHDRLBXEAB3VTU6IEPCAWQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.487859041 (104h10m25.084630054s) [noise=true , address=64.49.13.139:4474 , piece_id=HFVZT3V5KD42X64HCTKPEG2LWZJZAVOH5MA5CFCIMPB7WK7MWQNQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:27.489472801 (104h10m25.086243832s) [noise=true , address=64.49.13.170:4725 , piece_id=DEFHEGRC3SUXYVLYV2YR3KPBS4GLCPQBMV2GWR7KRY4ZBWDGAJJA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:10:28.942749218 (104h10m26.539520243s) [noise=true , address=79.127.205.237:4245 , piece_id=ZO6RJQU6Z5XIV6NXYBV7PWACGAEP7PWJMA7VISBKQS7IRWOXNRZQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: <nil> Mar 10 19:11:26.775681128 (104h11m24.37245216s) [noise=true , address=77.90.31.17:4793 , piece_id=GB6MI5XNLXQL4KFADPIKGKDKIQ7LNZ65EOTKY7LLSQMYY7WS56FQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775748621 (104h11m24.37251965s) [noise=true , address=77.90.31.12:4604 , piece_id=JUIZARJIP27O4PO3IZ54Y734TUIG2I3DUCQNFWI5TH5ZMCBDGJMA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: rpc: dial tcp 77.90.31.12:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.775972253 (104h11m24.372743279s) [noise=true , address=77.90.31.16:4754 , piece_id=PE4KKNG6B7T5FTCJOYIKG2ML2XOAD6SYAKAIRTKSKQM66GVTFASQ, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: write tcp 159.223.143.39:58400->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776001817 (104h11m24.37277285s) [noise=true , address=77.90.31.14:4690 , piece_id=XUIROGKLTQGEBWNIYKM5LA6U4DZG3ZYFMCHMNYCZJ2R4Y5WB7Z5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776244713 (104h11m24.373015806s) [noise=true , address=77.90.31.18:4814 , piece_id=OUCWOBYCYXTPTVHHOOILTZTLUB6PJ42HOV3YURYC3LTKMTDDGPSA, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: write tcp 159.223.143.39:36246->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:11:26.776280556 (104h11m24.373051587s) [noise=true , address=77.90.31.11:4585 , piece_id=VX7QB2HCS64PW5BIRIJ3RUY7V5FMDVTO423ENNPOTSQUINRR6R5A, seg_pos={0 0}, upload=116599]: Done uploading piece. err: ecclient: upload failed (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG, address:77.90.31.11:4585): protocol: context deadline exceeded; context deadline exceeded Mar 10 19:11:26.776313694 (104h11m24.373084705s) [seg_pos={0 0}, upload=116599]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 19:11:26.804828285 (104h11m24.401599298s) [upload=116599]: Done uploading segments... Mar 10 19:11:26.805017989 (104h11m24.401789013s) [upload=116599]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.031826192s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X..................X.....................................] duration: 814.392726ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:11:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 814.484426ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [................................................X......X....] duration: 7.414559729s trace: https://jaeger.dev.storj.io/trace/60e038fc0a6bf5b6 FAIL (2026-03-10 19:11:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== pload=116536]: Uploading piece... Mar 10 19:07:08.052812464 (104h7m5.649583488s) [noise=true , address=89.149.201.147:4012 , piece_id=6FCDV7M3V6OYZNCRXFKU7AZFQOIOUDCTRXZFZRVKY5D653CVVSRQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053192041 (104h7m5.649963064s) [noise=true , address=185.167.204.206:4053 , piece_id=FL3WYO7QV7WVPNTC5L2HQIZLZ5Y5KXCCCHTFZ4WLMHLYY2DG3RXQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053534479 (104h7m5.650305505s) [noise=true , address=81.171.14.70:4030 , piece_id=CSNUZOIPMFMJWTOFAWBPXSSET7ITOX4XHSVP5OPQQAQOTFCHOJ4A, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053903216 (104h7m5.650674239s) [noise=true , address=185.167.204.206:4065 , piece_id=73HCX24X7GU4EXZHRDM7BV6UNKA5FNYVCNHWXIQXW7OV4VETWDNA, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.054577271 (104h7m5.651348304s) [noise=true , address=185.167.204.206:4049 , piece_id=NI52TPUMI4JVQ5TVMNQKN6IW3YZSU4EBXCLECA4PP7YSPVSNMTZQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.054916818 (104h7m5.651687833s) [noise=true , address=81.171.14.70:4039 , piece_id=B5KA4MB6AQTBQIGJBRHR2BVUOMRH5ISISW6GRTCCQ7KVDYGXWOYA, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055013708 (104h7m5.651784734s) [noise=true , address=185.167.204.206:4047 , piece_id=2LYHINYE53WQOF34OLI2KA4HUNEBU2SADVHQFMZBVZB4MPSAV6RQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055398681 (104h7m5.652169696s) [noise=true , address=89.149.201.147:4014 , piece_id=2CDEUE6S3DFXJ57KQIAO7ZUTBXBZLL7ORFR2SJ5PLGGC7WSXCS5Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055413527 (104h7m5.652184552s) [noise=true , address=81.171.14.70:4031 , piece_id=22JBHNL7OQRH4TOBPKYIFH5ZVIIL2LIJDFQGGLXBW6UGTDBC4EBQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055769117 (104h7m5.652540146s) [noise=true , address=81.171.14.70:4026 , piece_id=CVYTYXF7RPZS4FMUPFUMINXJBXY42NOI3QHI5A5YSU26FTSGA72Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055793969 (104h7m5.652564986s) [noise=true , address=89.149.201.147:4013 , piece_id=T2VCAXBO5MJW5THHW7AOSSZJ3FJUV3WCVMMHSAONG7VXY6OCSZ2Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.059615569 (104h7m5.656386589s) [noise=true , address=89.149.201.147:4007 , piece_id=ICSU7CLO5VDMAJSX47CCMF3ZCHPGEZT6TWV3PTD25WCWB4MCO7JQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.483435998 (104h7m6.080207019s) [upload=116536]: Next returned nil segment Mar 10 19:07:08.483463545 (104h7m6.080234571s) [upload=116536]: Waiting for error group managing segments... Mar 10 19:07:08.670085108 (104h7m6.266856132s) [noise=true , address=81.171.14.70:4030 , piece_id=CSNUZOIPMFMJWTOFAWBPXSSET7ITOX4XHSVP5OPQQAQOTFCHOJ4A, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.671751688 (104h7m6.26852271s) [noise=true , address=81.171.14.70:4039 , piece_id=B5KA4MB6AQTBQIGJBRHR2BVUOMRH5ISISW6GRTCCQ7KVDYGXWOYA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.675713790 (104h7m6.272484814s) [noise=true , address=81.171.14.70:4026 , piece_id=CVYTYXF7RPZS4FMUPFUMINXJBXY42NOI3QHI5A5YSU26FTSGA72Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.678729994 (104h7m6.275501017s) [noise=true , address=81.171.14.70:4031 , piece_id=22JBHNL7OQRH4TOBPKYIFH5ZVIIL2LIJDFQGGLXBW6UGTDBC4EBQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.679660985 (104h7m6.276432008s) [noise=true , address=185.167.204.206:4056 , piece_id=QNKSFJNUWFOKEEO4TBMWZQSMCABV3SES7XMQOBLIRB3Y4IEDNUZQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.704364612 (104h7m6.301135641s) [noise=true , address=89.149.201.147:4013 , piece_id=T2VCAXBO5MJW5THHW7AOSSZJ3FJUV3WCVMMHSAONG7VXY6OCSZ2Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.713873267 (104h7m6.310644294s) [noise=true , address=81.171.14.70:4036 , piece_id=EER6RSAHNMHEVN4XAX32OY3FUI4OK24I5LUFLPYHWDS34SNGPQRQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.716489526 (104h7m6.313260548s) [noise=true , address=185.167.204.205:4013 , piece_id=DLV3JPA46SEE2MLT2OUCYTTGRKE3YIFQ7J53SKCTKMZYDFRLQOVQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.720880819 (104h7m6.317651844s) [noise=true , address=185.167.204.206:4065 , piece_id=73HCX24X7GU4EXZHRDM7BV6UNKA5FNYVCNHWXIQXW7OV4VETWDNA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.721091989 (104h7m6.317863022s) [noise=true , address=81.171.14.70:4027 , piece_id=XUI3PS252PXEW6RU2KMS6JKB3VRWFAP4AH4MV2VHFNA2EY6BIG5A, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.722224062 (104h7m6.318995085s) [noise=true , address=89.149.201.147:4019 , piece_id=55XGKBTVO53PNYUPYO4AYAA7RQXNVCKHSB7VJ5FT474JJEOPIMCQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.726774177 (104h7m6.323545218s) [noise=true , address=89.149.201.147:4006 , piece_id=4275N735EX3ESSBGQMG6UURDYKZQT4KDTGI3UTZFPLZNUKMRBXEQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.740457541 (104h7m6.337228557s) [noise=true , address=185.167.204.206:4053 , piece_id=FL3WYO7QV7WVPNTC5L2HQIZLZ5Y5KXCCCHTFZ4WLMHLYY2DG3RXQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.764058724 (104h7m6.360829747s) [noise=true , address=185.167.204.206:4049 , piece_id=NI52TPUMI4JVQ5TVMNQKN6IW3YZSU4EBXCLECA4PP7YSPVSNMTZQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.764920220 (104h7m6.361691243s) [noise=true , address=185.167.204.206:4047 , piece_id=2LYHINYE53WQOF34OLI2KA4HUNEBU2SADVHQFMZBVZB4MPSAV6RQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.777059183 (104h7m6.373830232s) [noise=true , address=185.167.204.205:4005 , piece_id=WG55KC6CHZZTDQGQX4CKLT5IQN3PZS5HST6ADPV3K7IURYYMOPFQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.872009693 (104h7m6.468780719s) [noise=true , address=89.149.201.147:4012 , piece_id=6FCDV7M3V6OYZNCRXFKU7AZFQOIOUDCTRXZFZRVKY5D653CVVSRQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.184413047 (104h7m6.781184061s) [noise=true , address=81.171.14.70:4025 , piece_id=Q4ZIYTW5CSVBRXLFZFBUUZHBC4ER63OE7CNPZP5UBLNS2AFFT7HQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.218611147 (104h7m6.81538216s) [noise=true , address=81.171.14.70:4028 , piece_id=W2MHTUKXME6LLSBOCLUJWQAGEZM34ITWB2QVSO3NQWUFYIUSEY3Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.231788290 (104h7m6.828559319s) [noise=true , address=89.149.201.147:4024 , piece_id=SM35AGGVT24VDC46LEL5SR6XE3Y4WVEWFQJSFZS2K2XRMF64IWQA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.243351966 (104h7m6.840122993s) [noise=true , address=185.167.204.206:4045 , piece_id=KNNFTU6BBNDSTJPWTEKJIIYZQLRYL3T7SBSTFUUYI6EGPL2LFFBA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.911871668 (104h7m7.508642693s) [noise=true , address=185.167.204.205:4012 , piece_id=4G3LH5G6IGVEI3APRDKIYMEI7BT5H7MQYHXOVCLWAZZD433BV5YQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.957977058 (104h7m7.554748082s) [noise=true , address=185.167.204.206:4039 , piece_id=KC66HUYTZO4HBTMTPZF7OBPGEO2NVOHN6MXA256LQAOYM36AOSNQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:10.427584748 (104h7m8.024355769s) [noise=true , address=185.167.204.205:4001 , piece_id=WFICPLSRPVAWA2PTJGUP32TMVADLZ4M3E2YAOF64FJQFJBT2ATCQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:22.545770375 (104h7m20.142541405s) [noise=true , address=185.167.204.205:4019 , piece_id=7I4GCV5ACILSRHAUTT2DUPWORYIZAAMLGXQEDDSQWXWO66I3N7ZA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:08:07.676830562 (104h8m5.273601595s) [noise=true , address=89.149.201.147:4007 , piece_id=ICSU7CLO5VDMAJSX47CCMF3ZCHPGEZT6TWV3PTD25WCWB4MCO7JQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:38552->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.676856227 (104h8m5.273627257s) [noise=true , address=89.149.201.147:4021 , piece_id=2IYNAWS3HJWU7Z53DQ77BPWUXWA2W26Q5PTKUMLNXUGNWRK7NDLQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:35864->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.677137384 (104h8m5.273908417s) [noise=true , address=89.149.201.147:4014 , piece_id=2CDEUE6S3DFXJ57KQIAO7ZUTBXBZLL7ORFR2SJ5PLGGC7WSXCS5Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: read tcp 159.223.143.39:48614->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.677172053 (104h8m5.273943066s) [seg_pos={0 0}, upload=116536]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:08:07.777852887 (104h8m5.3746239s) [upload=116536]: Done uploading segments... Mar 10 19:08:07.777889836 (104h8m5.37466086s) [upload=116536]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 7.414593957s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X..................X....................................] duration: 654.378758ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:10:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 654.410202ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X.......X.............X....X..........X.....] duration: 1.110862039s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:10:38 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.111003236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 3 out of 60 is failed (threshold=3) [X................................................X......X...] duration: 17.498376385s trace: https://jaeger.dev.storj.io/trace/60e038fc0a6bf5b6 FAIL (2026-03-10 19:10:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== pload=116536]: Uploading piece... Mar 10 19:07:08.052812464 (104h7m5.649583488s) [noise=true , address=89.149.201.147:4012 , piece_id=6FCDV7M3V6OYZNCRXFKU7AZFQOIOUDCTRXZFZRVKY5D653CVVSRQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053192041 (104h7m5.649963064s) [noise=true , address=185.167.204.206:4053 , piece_id=FL3WYO7QV7WVPNTC5L2HQIZLZ5Y5KXCCCHTFZ4WLMHLYY2DG3RXQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053534479 (104h7m5.650305505s) [noise=true , address=81.171.14.70:4030 , piece_id=CSNUZOIPMFMJWTOFAWBPXSSET7ITOX4XHSVP5OPQQAQOTFCHOJ4A, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.053903216 (104h7m5.650674239s) [noise=true , address=185.167.204.206:4065 , piece_id=73HCX24X7GU4EXZHRDM7BV6UNKA5FNYVCNHWXIQXW7OV4VETWDNA, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.054577271 (104h7m5.651348304s) [noise=true , address=185.167.204.206:4049 , piece_id=NI52TPUMI4JVQ5TVMNQKN6IW3YZSU4EBXCLECA4PP7YSPVSNMTZQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.054916818 (104h7m5.651687833s) [noise=true , address=81.171.14.70:4039 , piece_id=B5KA4MB6AQTBQIGJBRHR2BVUOMRH5ISISW6GRTCCQ7KVDYGXWOYA, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055013708 (104h7m5.651784734s) [noise=true , address=185.167.204.206:4047 , piece_id=2LYHINYE53WQOF34OLI2KA4HUNEBU2SADVHQFMZBVZB4MPSAV6RQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055398681 (104h7m5.652169696s) [noise=true , address=89.149.201.147:4014 , piece_id=2CDEUE6S3DFXJ57KQIAO7ZUTBXBZLL7ORFR2SJ5PLGGC7WSXCS5Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055413527 (104h7m5.652184552s) [noise=true , address=81.171.14.70:4031 , piece_id=22JBHNL7OQRH4TOBPKYIFH5ZVIIL2LIJDFQGGLXBW6UGTDBC4EBQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055769117 (104h7m5.652540146s) [noise=true , address=81.171.14.70:4026 , piece_id=CVYTYXF7RPZS4FMUPFUMINXJBXY42NOI3QHI5A5YSU26FTSGA72Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.055793969 (104h7m5.652564986s) [noise=true , address=89.149.201.147:4013 , piece_id=T2VCAXBO5MJW5THHW7AOSSZJ3FJUV3WCVMMHSAONG7VXY6OCSZ2Q, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.059615569 (104h7m5.656386589s) [noise=true , address=89.149.201.147:4007 , piece_id=ICSU7CLO5VDMAJSX47CCMF3ZCHPGEZT6TWV3PTD25WCWB4MCO7JQ, seg_pos={0 0}, upload=116536]: Uploading piece... Mar 10 19:07:08.483435998 (104h7m6.080207019s) [upload=116536]: Next returned nil segment Mar 10 19:07:08.483463545 (104h7m6.080234571s) [upload=116536]: Waiting for error group managing segments... Mar 10 19:07:08.670085108 (104h7m6.266856132s) [noise=true , address=81.171.14.70:4030 , piece_id=CSNUZOIPMFMJWTOFAWBPXSSET7ITOX4XHSVP5OPQQAQOTFCHOJ4A, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.671751688 (104h7m6.26852271s) [noise=true , address=81.171.14.70:4039 , piece_id=B5KA4MB6AQTBQIGJBRHR2BVUOMRH5ISISW6GRTCCQ7KVDYGXWOYA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.675713790 (104h7m6.272484814s) [noise=true , address=81.171.14.70:4026 , piece_id=CVYTYXF7RPZS4FMUPFUMINXJBXY42NOI3QHI5A5YSU26FTSGA72Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.678729994 (104h7m6.275501017s) [noise=true , address=81.171.14.70:4031 , piece_id=22JBHNL7OQRH4TOBPKYIFH5ZVIIL2LIJDFQGGLXBW6UGTDBC4EBQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.679660985 (104h7m6.276432008s) [noise=true , address=185.167.204.206:4056 , piece_id=QNKSFJNUWFOKEEO4TBMWZQSMCABV3SES7XMQOBLIRB3Y4IEDNUZQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.704364612 (104h7m6.301135641s) [noise=true , address=89.149.201.147:4013 , piece_id=T2VCAXBO5MJW5THHW7AOSSZJ3FJUV3WCVMMHSAONG7VXY6OCSZ2Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.713873267 (104h7m6.310644294s) [noise=true , address=81.171.14.70:4036 , piece_id=EER6RSAHNMHEVN4XAX32OY3FUI4OK24I5LUFLPYHWDS34SNGPQRQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.716489526 (104h7m6.313260548s) [noise=true , address=185.167.204.205:4013 , piece_id=DLV3JPA46SEE2MLT2OUCYTTGRKE3YIFQ7J53SKCTKMZYDFRLQOVQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.720880819 (104h7m6.317651844s) [noise=true , address=185.167.204.206:4065 , piece_id=73HCX24X7GU4EXZHRDM7BV6UNKA5FNYVCNHWXIQXW7OV4VETWDNA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.721091989 (104h7m6.317863022s) [noise=true , address=81.171.14.70:4027 , piece_id=XUI3PS252PXEW6RU2KMS6JKB3VRWFAP4AH4MV2VHFNA2EY6BIG5A, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.722224062 (104h7m6.318995085s) [noise=true , address=89.149.201.147:4019 , piece_id=55XGKBTVO53PNYUPYO4AYAA7RQXNVCKHSB7VJ5FT474JJEOPIMCQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.726774177 (104h7m6.323545218s) [noise=true , address=89.149.201.147:4006 , piece_id=4275N735EX3ESSBGQMG6UURDYKZQT4KDTGI3UTZFPLZNUKMRBXEQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.740457541 (104h7m6.337228557s) [noise=true , address=185.167.204.206:4053 , piece_id=FL3WYO7QV7WVPNTC5L2HQIZLZ5Y5KXCCCHTFZ4WLMHLYY2DG3RXQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.764058724 (104h7m6.360829747s) [noise=true , address=185.167.204.206:4049 , piece_id=NI52TPUMI4JVQ5TVMNQKN6IW3YZSU4EBXCLECA4PP7YSPVSNMTZQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.764920220 (104h7m6.361691243s) [noise=true , address=185.167.204.206:4047 , piece_id=2LYHINYE53WQOF34OLI2KA4HUNEBU2SADVHQFMZBVZB4MPSAV6RQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.777059183 (104h7m6.373830232s) [noise=true , address=185.167.204.205:4005 , piece_id=WG55KC6CHZZTDQGQX4CKLT5IQN3PZS5HST6ADPV3K7IURYYMOPFQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:08.872009693 (104h7m6.468780719s) [noise=true , address=89.149.201.147:4012 , piece_id=6FCDV7M3V6OYZNCRXFKU7AZFQOIOUDCTRXZFZRVKY5D653CVVSRQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.184413047 (104h7m6.781184061s) [noise=true , address=81.171.14.70:4025 , piece_id=Q4ZIYTW5CSVBRXLFZFBUUZHBC4ER63OE7CNPZP5UBLNS2AFFT7HQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.218611147 (104h7m6.81538216s) [noise=true , address=81.171.14.70:4028 , piece_id=W2MHTUKXME6LLSBOCLUJWQAGEZM34ITWB2QVSO3NQWUFYIUSEY3Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.231788290 (104h7m6.828559319s) [noise=true , address=89.149.201.147:4024 , piece_id=SM35AGGVT24VDC46LEL5SR6XE3Y4WVEWFQJSFZS2K2XRMF64IWQA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.243351966 (104h7m6.840122993s) [noise=true , address=185.167.204.206:4045 , piece_id=KNNFTU6BBNDSTJPWTEKJIIYZQLRYL3T7SBSTFUUYI6EGPL2LFFBA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.911871668 (104h7m7.508642693s) [noise=true , address=185.167.204.205:4012 , piece_id=4G3LH5G6IGVEI3APRDKIYMEI7BT5H7MQYHXOVCLWAZZD433BV5YQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:09.957977058 (104h7m7.554748082s) [noise=true , address=185.167.204.206:4039 , piece_id=KC66HUYTZO4HBTMTPZF7OBPGEO2NVOHN6MXA256LQAOYM36AOSNQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:10.427584748 (104h7m8.024355769s) [noise=true , address=185.167.204.205:4001 , piece_id=WFICPLSRPVAWA2PTJGUP32TMVADLZ4M3E2YAOF64FJQFJBT2ATCQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:07:22.545770375 (104h7m20.142541405s) [noise=true , address=185.167.204.205:4019 , piece_id=7I4GCV5ACILSRHAUTT2DUPWORYIZAAMLGXQEDDSQWXWO66I3N7ZA, seg_pos={0 0}, upload=116536]: Done uploading piece. err: <nil> Mar 10 19:08:07.676830562 (104h8m5.273601595s) [noise=true , address=89.149.201.147:4007 , piece_id=ICSU7CLO5VDMAJSX47CCMF3ZCHPGEZT6TWV3PTD25WCWB4MCO7JQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew, address:89.149.201.147:4007): protocol: read tcp 159.223.143.39:38552->89.149.201.147:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.676856227 (104h8m5.273627257s) [noise=true , address=89.149.201.147:4021 , piece_id=2IYNAWS3HJWU7Z53DQ77BPWUXWA2W26Q5PTKUMLNXUGNWRK7NDLQ, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz, address:89.149.201.147:4021): protocol: read tcp 159.223.143.39:35864->89.149.201.147:4021: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.677137384 (104h8m5.273908417s) [noise=true , address=89.149.201.147:4014 , piece_id=2CDEUE6S3DFXJ57KQIAO7ZUTBXBZLL7ORFR2SJ5PLGGC7WSXCS5Q, seg_pos={0 0}, upload=116536]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: read tcp 159.223.143.39:48614->89.149.201.147:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:08:07.677172053 (104h8m5.273943066s) [seg_pos={0 0}, upload=116536]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Mar 10 19:08:07.777852887 (104h8m5.3746239s) [upload=116536]: Done uploading segments... Mar 10 19:08:07.777889836 (104h8m5.37466086s) [upload=116536]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 17.49844353s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X.......X.............X....X..........X....] duration: 1.43868603s trace: https://jaeger.dev.storj.io/trace/21780b32410ce66 FAIL (2026-03-10 19:10:13 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 116529]: Got next segment. Inline: false Mar 10 19:06:47.907812827 (104h6m45.504583852s) [upload=116529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:06:47.907821925 (104h6m45.504592955s) [upload=116529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:06:47.940557630 (104h6m45.537328665s) [seg_pos={0 0}, upload=116529]: Begin upload segment... Mar 10 19:06:47.940626826 (104h6m45.537397872s) [seg_pos={0 0}, upload=116529]: Done begin upload segment. Mar 10 19:06:47.940664974 (104h6m45.537436009s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.940762459 (104h6m45.537533493s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941373070 (104h6m45.538144102s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.941854421 (104h6m45.538625449s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.942384626 (104h6m45.539155657s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943308667 (104h6m45.5400797s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.943609163 (104h6m45.540380198s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944346996 (104h6m45.54111803s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.944371169 (104h6m45.541142202s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.945840127 (104h6m45.54261116s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.946047960 (104h6m45.542818985s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947317100 (104h6m45.544088132s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.947712870 (104h6m45.544483904s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948608160 (104h6m45.545379198s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.948926354 (104h6m45.545697388s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.949957787 (104h6m45.546728825s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951273293 (104h6m45.548044327s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:47.951829900 (104h6m45.548600934s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Uploading piece... Mar 10 19:06:48.036482492 (104h6m45.633253551s) [upload=116529]: Next returned nil segment Mar 10 19:06:48.036501010 (104h6m45.633272043s) [upload=116529]: Waiting for error group managing segments... Mar 10 19:06:48.042266571 (104h6m45.639037584s) [noise=true , address=45.91.116.14:4222 , piece_id=DN3SNW3LKHWHUL6B3PAS5P7OG44ZDWKQXKKOPU53LJDSKXKNBREQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.043683514 (104h6m45.640454543s) [noise=true , address=145.79.227.21:4483 , piece_id=7MSS7NE4SUWLFOU7KSAGCZ3IZVIJNSE56KHGRL3GCGQVJ3DKP4VQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.044327860 (104h6m45.641098874s) [noise=true , address=79.127.219.45:4881 , piece_id=K6IU7ZLP5S7YK6SPIRSQH4EDSPDS57MB3T5FPMVIL6BEH257LM7Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.046521721 (104h6m45.643292745s) [noise=true , address=79.127.219.41:4937 , piece_id=TB2LRED4R7QC7R2QMD5ZGRC6236JKDGO7KZLDKCC4K7M676YPQMQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.049264342 (104h6m45.646035356s) [noise=true , address=45.91.116.13:4170 , piece_id=R56YVTNBMRBVVY4J4JMTLQSJJ46DVY4CRQM6RIA2YNGA3CKPR7UQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.051826600 (104h6m45.648597629s) [noise=true , address=45.91.116.8:4068 , piece_id=Q6ERVQ3EO4ET54DDXKWHZWENPRY7FDEINMK5RVFT2YSFFPN3WVEQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.053239446 (104h6m45.65001047s) [noise=true , address=45.91.116.5:4308 , piece_id=Y4K77UYELRJEJAXBMBYBJ3SUJ7BNEOQ4NUCPLXWY24OBMVJIJWTA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.057554590 (104h6m45.654325622s) [noise=true , address=45.91.116.6:4978 , piece_id=TCYHWUUEZNKGOGJ3OSHOJU7I5SXSO7NODMUFAF5HN67GYS66RRWA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.347909714 (104h6m45.944680732s) [noise=true , address=167.17.72.71:4811 , piece_id=YK6OJXQTR5AIHI5CYQYRRAU4UXGC4LHHEOK56UG4OKDKT3TI2UVA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.497098288 (104h6m46.093869316s) [noise=true , address=79.127.205.234:4181 , piece_id=UD6B44BMN7H2WK3P2TIIAUFKZB2K6PH3EEIFBAVFED2Q53TQDDZQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.524549543 (104h6m46.121320561s) [noise=true , address=64.49.13.154:4928 , piece_id=VQSF7TC2IYV4S52MG3ID7PBEMPHL27JARON4XDJWJK4TOX2DJ6RA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:48.567475047 (104h6m46.164246061s) [noise=true , address=64.49.13.138:4487 , piece_id=FOCEUSC64R5Q7ZOCWE5XKAZXSQDXLDUEQ6TACPWYY2PXRUR4LJZA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:06:49.393957509 (104h6m46.990728537s) [noise=true , address=79.127.205.237:4234 , piece_id=7HTPOWTGA2GFYYSNDOOB5PNE44GPJKYVCZCEVFOUL24GJMJJC6EA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: <nil> Mar 10 19:07:47.865864834 (104h7m45.462635862s) [noise=true , address=77.90.31.19:4848 , piece_id=M4SDGI2RIF2TAUL3YQAP5AQ345ZKHQYV4CGPXC4KKM55FU6TNSPQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:35766->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866066060 (104h7m45.462837092s) [noise=true , address=77.90.31.15:4713 , piece_id=7FAXWQKYDTYUAAZGKIVFZC4534BJDJ4SEJFBJO335WCRSDRUHZAQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866185581 (104h7m45.462956607s) [noise=true , address=77.90.31.18:4812 , piece_id=GQUHTOY4SM5B2GRHCAFZEG5W5HKRXM3R42EHBDZVFPZC2DWG3WUA, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: write tcp 159.223.143.39:44376->77.90.31.18:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866263516 (104h7m45.463034544s) [noise=true , address=77.90.31.10:4535 , piece_id=AQ3Z632XLGXGOTGTZUIVK53Q5NXGFMWCVYYLOHTJWURBAP6HQK4Q, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:48368->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866517656 (104h7m45.463288691s) [noise=true , address=77.90.31.17:4786 , piece_id=FJJFNH5LLXEJ3AU3O2WLI5F7TU6LESJZQY72Q7O27QLRK5KOVLNQ, seg_pos={0 0}, upload=116529]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:40520->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:07:47.866574294 (104h7m45.463345325s) [seg_pos={0 0}, upload=116529]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:07:47.888227569 (104h7m45.484998602s) [upload=116529]: Done uploading segments... Mar 10 19:07:47.888332065 (104h7m45.485103168s) [upload=116529]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.438734472s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X..................X...................................] duration: 795.063177ms trace: https://jaeger.dev.storj.io/trace/66877fbb536219fd FAIL (2026-03-10 19:10:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 795.092008ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 759.147116ms trace: https://jaeger.dev.storj.io/trace/33e9eb6cbded500c FAIL (2026-03-10 19:10:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 759.180468ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)