Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.042809915s FAIL (2025-07-26 21:56:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== WCNEOQ, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.180011702 (3m46.960945097s) [noise=true , address=104.36.82.46:4508 , piece_id=L7CBK727HEW3OOCD6QZA3SMIIABUT7NM74PBTIO4LU5YZ35B7NQA, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.180466513 (3m46.9613999s) [noise=true , address=104.36.82.56:4244 , piece_id=JNMIC5SUQADYNIH4G5NPHPLRLP3JVSN45SACXZL3EI2UC4I3AIDQ, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.180542547 (3m46.96147593s) [noise=true , address=104.36.82.16:4300 , piece_id=NX7GNFMKZ6OV6FDGYNVZ3QRNPI3EOMEVBWXB6IYZ3GWW7KXHSAUQ, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.180914968 (3m46.961848352s) [noise=true , address=79.127.205.246:4282 , piece_id=G6GXXADZC7J2XXN3NTDU47YBGXOUF6ASJ554SPWCELTMRJS47FEQ, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.180965959 (3m46.961899346s) [noise=true , address=104.36.82.52:4102 , piece_id=4XSU4Y4CEEPYH7FWH6QFA2E5GK5RXBJB2Y3WFZL4FXFK2H3FBIVQ, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.181328296 (3m46.962261683s) [noise=true , address=156.146.43.225:4448 , piece_id=SDKMWK62K4DMIUXCTHNP4AQHVN5WKXTTZROJWUWVRMX7OEEO2LWA, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.181399386 (3m46.962332772s) [noise=true , address=104.36.82.47:4531 , piece_id=5VDLQ677J7EWAJZI3EF7ZA5Z3GOZ7HF7H3PIIWOI22KG45LEBVOQ, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.181669708 (3m46.962603088s) [noise=true , address=79.127.205.233:4643 , piece_id=EWAU2RZHRFTI7KHNWTDXZUKPMRNZQZWRADKNTD5CVUGYZR2YASDA, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.181919462 (3m46.96285285s) [noise=true , address=79.127.201.215:4350 , piece_id=E2ZHBYLBYGEHRXJ6XRY5ZHHRG2Z4DW7TEVKECPGJ4IHDSNYB6SKQ, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.182456386 (3m46.96338977s) [noise=true , address=104.36.82.3:4784 , piece_id=FYGXCM6DG6XGGCKB3LHB4MZSCAXBUQ7P6H5THU5ANVEYJ5VDUF5A, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.183253654 (3m46.964187034s) [noise=true , address=104.36.82.3:4785 , piece_id=RNVOLYT7MNLXMRZGQLNLUE232FXN7IILBLCFDLKEDJIFVL3OL3OA, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.184979871 (3m46.965913254s) [noise=true , address=104.36.82.45:4935 , piece_id=26HIZ5Y7RY36K3AP37DQQFRO4W3DAMQ7RJCZRLWMGQYF4T5ZROMA, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.185311884 (3m46.966245267s) [noise=true , address=104.36.82.9:4011 , piece_id=YBXFAMO5PBOJ7IEWBI6YN7QW3FIA6TM2KVFYPANZWWQ5WUD3KCAQ, seg_pos={0 0}, upload=53]: Uploading piece... Jul 26 21:55:31.385863754 (3m47.166797119s) [upload=53]: Next returned nil segment Jul 26 21:55:31.385883656 (3m47.166817044s) [upload=53]: Waiting for error group managing segments... Jul 26 21:55:31.807163460 (3m47.588096842s) [noise=true , address=79.127.205.233:4643 , piece_id=EWAU2RZHRFTI7KHNWTDXZUKPMRNZQZWRADKNTD5CVUGYZR2YASDA, seg_pos={0 0}, upload=53]: Done uploading piece. err: <nil> Jul 26 21:55:31.830655326 (3m47.611588693s) [noise=true , address=79.127.205.246:4282 , piece_id=G6GXXADZC7J2XXN3NTDU47YBGXOUF6ASJ554SPWCELTMRJS47FEQ, seg_pos={0 0}, upload=53]: Done uploading piece. err: <nil> Jul 26 21:55:32.300660545 (3m48.081593921s) [noise=true , address=156.146.43.225:4448 , piece_id=SDKMWK62K4DMIUXCTHNP4AQHVN5WKXTTZROJWUWVRMX7OEEO2LWA, seg_pos={0 0}, upload=53]: Done uploading piece. err: <nil> Jul 26 21:55:34.436809563 (3m50.217742926s) [noise=true , address=79.127.201.215:4350 , piece_id=E2ZHBYLBYGEHRXJ6XRY5ZHHRG2Z4DW7TEVKECPGJ4IHDSNYB6SKQ, seg_pos={0 0}, upload=53]: Done uploading piece. err: <nil> Jul 26 21:55:41.771909838 (3m57.552843229s) [noise=true , address=77.90.31.12:4608 , piece_id=NCGJJVPGSFBVJW5M2OAWH6P4SFD5ONZTPK2N2OWHKSPSTD3YV7MA, seg_pos={0 0}, upload=53]: Done uploading piece. err: <nil> Jul 26 21:56:31.112712381 (4m46.893645765s) [noise=true , address=104.36.82.41:4766 , piece_id=NVR7IQSHOUC5P44T5SZ4PYLE6L3CJQXO5DYUAO7MRXHLV4P5ECAA, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: rpc: dial tcp 104.36.82.41:4766: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.112842400 (4m46.893775776s) [noise=true , address=104.36.82.55:4227 , piece_id=MGNPS6VLTSDUMIF574AOYEHDQPYVDST3RI52OETEN7CDHRWCNEOQ, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:19qGGwWT9xUYsJyRnFHyHKPFr6g4AsckxRNVTg2shfVhoQtcCz, address:104.36.82.55:4227): protocol: rpc: dial tcp 104.36.82.55:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.112924553 (4m46.893857939s) [noise=true , address=104.36.82.56:4244 , piece_id=JNMIC5SUQADYNIH4G5NPHPLRLP3JVSN45SACXZL3EI2UC4I3AIDQ, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:12S1wh3PFQLPosjWmPs5iLfm4S5J6abXvoCfkhYz3KdvXCTaPRu, address:104.36.82.56:4244): protocol: rpc: dial tcp 104.36.82.56:4244: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.112997888 (4m46.893931265s) [noise=true , address=104.36.82.47:4531 , piece_id=5VDLQ677J7EWAJZI3EF7ZA5Z3GOZ7HF7H3PIIWOI22KG45LEBVOQ, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.113226696 (4m46.894160076s) [noise=true , address=104.36.82.10:4086 , piece_id=KRVBCOLZMKBUR7L7W7A2FBPMOOB3PFXTGRJAXEYXNGF3YFQ4K52Q, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:1fMyQ9ojAvrZhG4sLwG3wzNJZihg8ZwWKBJmgWpK3Ckx5YXrg3, address:104.36.82.10:4086): protocol: rpc: dial tcp 104.36.82.10:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.113299577 (4m46.894232953s) [noise=true , address=104.36.82.9:4011 , piece_id=YBXFAMO5PBOJ7IEWBI6YN7QW3FIA6TM2KVFYPANZWWQ5WUD3KCAQ, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:12J8J47emDh4p115m69EqUYroi9GgruC3y8gmv2xVULVJSvFYYP, address:104.36.82.9:4011): protocol: write tcp 159.223.143.39:53964->104.36.82.9:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.113465073 (4m46.894398455s) [noise=true , address=104.36.82.52:4102 , piece_id=4XSU4Y4CEEPYH7FWH6QFA2E5GK5RXBJB2Y3WFZL4FXFK2H3FBIVQ, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:12evG7WqJ9XXqh2z2kjGr4Wz1q5mrcySfTMczqEm7i8jgYsW8qP, address:104.36.82.52:4102): protocol: rpc: dial tcp 104.36.82.52:4102: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.114079558 (4m46.895012931s) [noise=true , address=104.36.82.11:4132 , piece_id=E24DZN3CXS2BT3WJ7AXCVEJH7QVG7CCEBHY5WWGIUHUMKDFNVCWA, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.114160964 (4m46.895094341s) [noise=true , address=104.36.82.46:4508 , piece_id=L7CBK727HEW3OOCD6QZA3SMIIABUT7NM74PBTIO4LU5YZ35B7NQA, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: write tcp 159.223.143.39:35986->104.36.82.46:4508: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.114230678 (4m46.895164058s) [noise=true , address=104.36.82.3:4784 , piece_id=FYGXCM6DG6XGGCKB3LHB4MZSCAXBUQ7P6H5THU5ANVEYJ5VDUF5A, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:1534abg2Jn4zTQzvBcDDZDLq2QZp6kBKY1q6zEbk2PC8mJrnq3, address:104.36.82.3:4784): protocol: write tcp 159.223.143.39:60434->104.36.82.3:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.114444856 (4m46.895378238s) [noise=true , address=104.36.82.3:4785 , piece_id=RNVOLYT7MNLXMRZGQLNLUE232FXN7IILBLCFDLKEDJIFVL3OL3OA, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:104.36.82.3:4785): protocol: rpc: dial tcp 104.36.82.3:4785: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.114463516 (4m46.895396891s) [noise=true , address=104.36.82.16:4300 , piece_id=NX7GNFMKZ6OV6FDGYNVZ3QRNPI3EOMEVBWXB6IYZ3GWW7KXHSAUQ, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg, address:104.36.82.16:4300): protocol: rpc: dial tcp 104.36.82.16:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.114526307 (4m46.895459684s) [noise=true , address=104.36.82.45:4935 , piece_id=26HIZ5Y7RY36K3AP37DQQFRO4W3DAMQ7RJCZRLWMGQYF4T5ZROMA, seg_pos={0 0}, upload=53]: Done uploading piece. err: ecclient: upload failed (node:12Ks2kQWUmV377HgGbyoTt96CTyK1P1nH8QtZQzh3xvnFCYaD8X, address:104.36.82.45:4935): protocol: rpc: dial tcp 104.36.82.45:4935: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:31.114552495 (4m46.89548588s) [seg_pos={0 0}, upload=53]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 21:56:31.151114324 (4m46.93204769s) [upload=53]: Done uploading segments... Jul 26 21:56:31.151168043 (4m46.932101419s) [upload=53]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.042833994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 6.219272646s FAIL (2025-07-26 21:56:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 2): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037373424 (3m35.818306814s) [noise=true , address=104.36.82.13:4186 , piece_id=3P4TVOZ2BRS2RJT2C6HWXK2ZDN5ZCHAQPWXSMPOD63653PUAZOOA, seg_pos={0 0}, upload=43]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037396511 (3m35.818329895s) [noise=true , address=104.36.82.14:4223 , piece_id=4F6ADV22WOQSN3HKFM4JCVZA3OWSYX2S6OP2TAH4SBOKHJ2AQCYQ, seg_pos={0 0}, upload=43]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:51250->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037425149 (3m35.818358546s) [seg_pos={0 0}, upload=43]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 21:55:20.065111780 (3m35.846045157s) [upload=43]: Done uploading segments... Jul 26 21:55:20.065145366 (3m35.846078742s) [upload=43]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 21:55:20.164756852 (3m35.945690235s) [upload=50]: Starting upload Jul 26 21:55:20.164893006 (3m35.94582639s) [upload=50]: Done starting upload Jul 26 21:55:20.165026534 (3m35.945959914s) [upload=50]: Uploading segments... Jul 26 21:55:20.165348430 (3m35.946281792s) [upload=50]: Got next segment. Inline: false Jul 26 21:55:20.165436135 (3m35.946369521s) [upload=50]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:55:20.165462754 (3m35.946396131s) [upload=50]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:55:20.199323680 (3m35.980257066s) [seg_pos={0 0}, upload=50]: Begin upload segment... Jul 26 21:55:20.199390208 (3m35.980323572s) [seg_pos={0 0}, upload=50]: Done begin upload segment. Jul 26 21:55:20.199432997 (3m35.980366388s) [noise=true , address=45.91.116.9:4263 , piece_id=HYALUZFPHNNMNH3OWJTFE5A2KKKI7VCHAY7D7MRRYZHFON4NRGNA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.199457515 (3m35.980390901s) [noise=true , address=45.91.116.8:4048 , piece_id=QRWTDVH75BD3AG4HUB5K3OCSTN67ZYNUA47R5HD26MABIYX6HSHA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.199933992 (3m35.98086738s) [noise=true , address=104.36.82.16:4294 , piece_id=LJC6J2ZGRGQCQCJTVRYKBITAEQDD4SWDVUXDIKH2ZJ6GHONHC4TQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200044523 (3m35.98097791s) [noise=true , address=77.90.31.10:4520 , piece_id=WVDWEFRRKQHEKOTCLLXLRFBOKNIIYP5A7UUG65D3DNDBC64BKKTQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200436225 (3m35.981369612s) [noise=true , address=77.90.31.18:4805 , piece_id=DLOTLJTQIB7MPNFFX6ZJZKYM2SH6DCK42QQ6JSX44OP5X6F6FR2Q, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200695832 (3m35.981629215s) [noise=true , address=45.91.116.5:4298 , piece_id=JGALBSMMJ2DFR5SKFIFKIIQT2SBK4ROHMOF7MLF644ISBDYFTURA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200822101 (3m35.98175549s) [noise=true , address=45.91.116.12:4225 , piece_id=CCMDTAPIALMUNTERMHI2L62BZLAPDSUNB6O2C4FJYNV4MQLQZLRQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.201228239 (3m35.982161626s) [noise=true , address=45.91.116.4:4937 , piece_id=3B23LIDMZHLGAYD26YFGI7UQVIE5THZJ2JOYUZMAALPAGVJGMHLA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.201625940 (3m35.982559326s) [noise=true , address=45.91.116.14:4189 , piece_id=K4EG4BAMVLHWI5JHJS2PBBLABJQ7X6OGIUVNDBIZ4RNUKEEI3AEA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.201947856 (3m35.982881236s) [noise=true , address=45.91.116.6:4973 , piece_id=CZRULLAJYSH6BTMT64ZTHOW34R53E5PEJWQLYNZELVL4QSIOEVCA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.202017752 (3m35.982951149s) [noise=true , address=104.36.82.4:4803 , piece_id=QTE2FUEJ4SUC3YBZH22CP26N7ABI7XLJUNJ4ZVLH7V2YUXZWJ23Q, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.202423768 (3m35.983357154s) [noise=true , address=45.91.116.5:4299 , piece_id=AVWBPHYL2RDCSCMYMHQBMYE6ILP52JLRINBZTTHX2VGZENSU7XIQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.202810152 (3m35.983743548s) [noise=true , address=45.91.116.11:4119 , piece_id=EYMY3OMR4DYV2LKRZLTFCEXHY4TBMZNHIMBNPJYRR7LXR6SO7VCQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.203207751 (3m35.984141134s) [noise=true , address=104.36.82.6:4875 , piece_id=MV5GXWFIFYAZIOXE7CW7XP3QBCFZSF3AJXOGKK5GVYPUS3FFJJVA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.203572200 (3m35.984505585s) [noise=true , address=45.91.116.10:4081 , piece_id=TWHMEGHDDMYNCTNXV4BJVXRB7YSI7XNITPUIR7KKNNGSV3EUTCQQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.203919318 (3m35.984852704s) [noise=true , address=104.36.82.7:4909 , piece_id=MH34BK5ZDXUZX5XUUFR6YWOMRD6CQB2UHPN7RS7DWCBGVLISXVIA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.305395509 (3m36.086328891s) [upload=50]: Next returned nil segment Jul 26 21:55:20.305415668 (3m36.08634905s) [upload=50]: Waiting for error group managing segments... Jul 26 21:55:20.312143340 (3m36.093076718s) [noise=true , address=45.91.116.9:4263 , piece_id=HYALUZFPHNNMNH3OWJTFE5A2KKKI7VCHAY7D7MRRYZHFON4NRGNA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.313065385 (3m36.093998763s) [noise=true , address=45.91.116.4:4937 , piece_id=3B23LIDMZHLGAYD26YFGI7UQVIE5THZJ2JOYUZMAALPAGVJGMHLA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.313873574 (3m36.094806954s) [noise=true , address=45.91.116.5:4299 , piece_id=AVWBPHYL2RDCSCMYMHQBMYE6ILP52JLRINBZTTHX2VGZENSU7XIQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.313911757 (3m36.094845139s) [noise=true , address=45.91.116.10:4081 , piece_id=TWHMEGHDDMYNCTNXV4BJVXRB7YSI7XNITPUIR7KKNNGSV3EUTCQQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.314560119 (3m36.095493488s) [noise=true , address=45.91.116.5:4298 , piece_id=JGALBSMMJ2DFR5SKFIFKIIQT2SBK4ROHMOF7MLF644ISBDYFTURA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.314625417 (3m36.095558796s) [noise=true , address=45.91.116.12:4225 , piece_id=CCMDTAPIALMUNTERMHI2L62BZLAPDSUNB6O2C4FJYNV4MQLQZLRQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315101138 (3m36.096034506s) [noise=true , address=45.91.116.8:4048 , piece_id=QRWTDVH75BD3AG4HUB5K3OCSTN67ZYNUA47R5HD26MABIYX6HSHA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315321282 (3m36.096254661s) [noise=true , address=45.91.116.11:4119 , piece_id=EYMY3OMR4DYV2LKRZLTFCEXHY4TBMZNHIMBNPJYRR7LXR6SO7VCQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315702554 (3m36.096635934s) [noise=true , address=45.91.116.6:4973 , piece_id=CZRULLAJYSH6BTMT64ZTHOW34R53E5PEJWQLYNZELVL4QSIOEVCA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315909249 (3m36.096842626s) [noise=true , address=45.91.116.14:4189 , piece_id=K4EG4BAMVLHWI5JHJS2PBBLABJQ7X6OGIUVNDBIZ4RNUKEEI3AEA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:25.353897443 (3m41.134830808s) [noise=true , address=77.90.31.18:4805 , piece_id=DLOTLJTQIB7MPNFFX6ZJZKYM2SH6DCK42QQ6JSX44OP5X6F6FR2Q, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:25.406007691 (3m41.186941069s) [noise=true , address=77.90.31.10:4520 , piece_id=WVDWEFRRKQHEKOTCLLXLRFBOKNIIYP5A7UUG65D3DNDBC64BKKTQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:50.132368988 (4m5.913302372s) [noise=true , address=104.36.82.16:4294 , piece_id=LJC6J2ZGRGQCQCJTVRYKBITAEQDD4SWDVUXDIKH2ZJ6GHONHC4TQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:50.132475169 (4m5.913408545s) [noise=true , address=104.36.82.4:4803 , piece_id=QTE2FUEJ4SUC3YBZH22CP26N7ABI7XLJUNJ4ZVLH7V2YUXZWJ23Q, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:50.132612953 (4m5.913546337s) [noise=true , address=104.36.82.7:4909 , piece_id=MH34BK5ZDXUZX5XUUFR6YWOMRD6CQB2UHPN7RS7DWCBGVLISXVIA, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:55:50.132844023 (4m5.913777387s) [noise=true , address=104.36.82.6:4875 , piece_id=MV5GXWFIFYAZIOXE7CW7XP3QBCFZSF3AJXOGKK5GVYPUS3FFJJVA, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:55:50.132873238 (4m5.913806624s) [seg_pos={0 0}, upload=50]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:55:50.156981397 (4m5.937914767s) [upload=50]: Done uploading segments... Jul 26 21:55:50.157007415 (4m5.93794079s) [upload=50]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.219297525s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.027306974s FAIL (2025-07-26 21:55:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 2): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037373424 (3m35.818306814s) [noise=true , address=104.36.82.13:4186 , piece_id=3P4TVOZ2BRS2RJT2C6HWXK2ZDN5ZCHAQPWXSMPOD63653PUAZOOA, seg_pos={0 0}, upload=43]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037396511 (3m35.818329895s) [noise=true , address=104.36.82.14:4223 , piece_id=4F6ADV22WOQSN3HKFM4JCVZA3OWSYX2S6OP2TAH4SBOKHJ2AQCYQ, seg_pos={0 0}, upload=43]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:51250->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037425149 (3m35.818358546s) [seg_pos={0 0}, upload=43]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 21:55:20.065111780 (3m35.846045157s) [upload=43]: Done uploading segments... Jul 26 21:55:20.065145366 (3m35.846078742s) [upload=43]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 21:55:20.164756852 (3m35.945690235s) [upload=50]: Starting upload Jul 26 21:55:20.164893006 (3m35.94582639s) [upload=50]: Done starting upload Jul 26 21:55:20.165026534 (3m35.945959914s) [upload=50]: Uploading segments... Jul 26 21:55:20.165348430 (3m35.946281792s) [upload=50]: Got next segment. Inline: false Jul 26 21:55:20.165436135 (3m35.946369521s) [upload=50]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:55:20.165462754 (3m35.946396131s) [upload=50]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:55:20.199323680 (3m35.980257066s) [seg_pos={0 0}, upload=50]: Begin upload segment... Jul 26 21:55:20.199390208 (3m35.980323572s) [seg_pos={0 0}, upload=50]: Done begin upload segment. Jul 26 21:55:20.199432997 (3m35.980366388s) [noise=true , address=45.91.116.9:4263 , piece_id=HYALUZFPHNNMNH3OWJTFE5A2KKKI7VCHAY7D7MRRYZHFON4NRGNA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.199457515 (3m35.980390901s) [noise=true , address=45.91.116.8:4048 , piece_id=QRWTDVH75BD3AG4HUB5K3OCSTN67ZYNUA47R5HD26MABIYX6HSHA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.199933992 (3m35.98086738s) [noise=true , address=104.36.82.16:4294 , piece_id=LJC6J2ZGRGQCQCJTVRYKBITAEQDD4SWDVUXDIKH2ZJ6GHONHC4TQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200044523 (3m35.98097791s) [noise=true , address=77.90.31.10:4520 , piece_id=WVDWEFRRKQHEKOTCLLXLRFBOKNIIYP5A7UUG65D3DNDBC64BKKTQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200436225 (3m35.981369612s) [noise=true , address=77.90.31.18:4805 , piece_id=DLOTLJTQIB7MPNFFX6ZJZKYM2SH6DCK42QQ6JSX44OP5X6F6FR2Q, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200695832 (3m35.981629215s) [noise=true , address=45.91.116.5:4298 , piece_id=JGALBSMMJ2DFR5SKFIFKIIQT2SBK4ROHMOF7MLF644ISBDYFTURA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.200822101 (3m35.98175549s) [noise=true , address=45.91.116.12:4225 , piece_id=CCMDTAPIALMUNTERMHI2L62BZLAPDSUNB6O2C4FJYNV4MQLQZLRQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.201228239 (3m35.982161626s) [noise=true , address=45.91.116.4:4937 , piece_id=3B23LIDMZHLGAYD26YFGI7UQVIE5THZJ2JOYUZMAALPAGVJGMHLA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.201625940 (3m35.982559326s) [noise=true , address=45.91.116.14:4189 , piece_id=K4EG4BAMVLHWI5JHJS2PBBLABJQ7X6OGIUVNDBIZ4RNUKEEI3AEA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.201947856 (3m35.982881236s) [noise=true , address=45.91.116.6:4973 , piece_id=CZRULLAJYSH6BTMT64ZTHOW34R53E5PEJWQLYNZELVL4QSIOEVCA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.202017752 (3m35.982951149s) [noise=true , address=104.36.82.4:4803 , piece_id=QTE2FUEJ4SUC3YBZH22CP26N7ABI7XLJUNJ4ZVLH7V2YUXZWJ23Q, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.202423768 (3m35.983357154s) [noise=true , address=45.91.116.5:4299 , piece_id=AVWBPHYL2RDCSCMYMHQBMYE6ILP52JLRINBZTTHX2VGZENSU7XIQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.202810152 (3m35.983743548s) [noise=true , address=45.91.116.11:4119 , piece_id=EYMY3OMR4DYV2LKRZLTFCEXHY4TBMZNHIMBNPJYRR7LXR6SO7VCQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.203207751 (3m35.984141134s) [noise=true , address=104.36.82.6:4875 , piece_id=MV5GXWFIFYAZIOXE7CW7XP3QBCFZSF3AJXOGKK5GVYPUS3FFJJVA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.203572200 (3m35.984505585s) [noise=true , address=45.91.116.10:4081 , piece_id=TWHMEGHDDMYNCTNXV4BJVXRB7YSI7XNITPUIR7KKNNGSV3EUTCQQ, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.203919318 (3m35.984852704s) [noise=true , address=104.36.82.7:4909 , piece_id=MH34BK5ZDXUZX5XUUFR6YWOMRD6CQB2UHPN7RS7DWCBGVLISXVIA, seg_pos={0 0}, upload=50]: Uploading piece... Jul 26 21:55:20.305395509 (3m36.086328891s) [upload=50]: Next returned nil segment Jul 26 21:55:20.305415668 (3m36.08634905s) [upload=50]: Waiting for error group managing segments... Jul 26 21:55:20.312143340 (3m36.093076718s) [noise=true , address=45.91.116.9:4263 , piece_id=HYALUZFPHNNMNH3OWJTFE5A2KKKI7VCHAY7D7MRRYZHFON4NRGNA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.313065385 (3m36.093998763s) [noise=true , address=45.91.116.4:4937 , piece_id=3B23LIDMZHLGAYD26YFGI7UQVIE5THZJ2JOYUZMAALPAGVJGMHLA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.313873574 (3m36.094806954s) [noise=true , address=45.91.116.5:4299 , piece_id=AVWBPHYL2RDCSCMYMHQBMYE6ILP52JLRINBZTTHX2VGZENSU7XIQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.313911757 (3m36.094845139s) [noise=true , address=45.91.116.10:4081 , piece_id=TWHMEGHDDMYNCTNXV4BJVXRB7YSI7XNITPUIR7KKNNGSV3EUTCQQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.314560119 (3m36.095493488s) [noise=true , address=45.91.116.5:4298 , piece_id=JGALBSMMJ2DFR5SKFIFKIIQT2SBK4ROHMOF7MLF644ISBDYFTURA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.314625417 (3m36.095558796s) [noise=true , address=45.91.116.12:4225 , piece_id=CCMDTAPIALMUNTERMHI2L62BZLAPDSUNB6O2C4FJYNV4MQLQZLRQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315101138 (3m36.096034506s) [noise=true , address=45.91.116.8:4048 , piece_id=QRWTDVH75BD3AG4HUB5K3OCSTN67ZYNUA47R5HD26MABIYX6HSHA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315321282 (3m36.096254661s) [noise=true , address=45.91.116.11:4119 , piece_id=EYMY3OMR4DYV2LKRZLTFCEXHY4TBMZNHIMBNPJYRR7LXR6SO7VCQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315702554 (3m36.096635934s) [noise=true , address=45.91.116.6:4973 , piece_id=CZRULLAJYSH6BTMT64ZTHOW34R53E5PEJWQLYNZELVL4QSIOEVCA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:20.315909249 (3m36.096842626s) [noise=true , address=45.91.116.14:4189 , piece_id=K4EG4BAMVLHWI5JHJS2PBBLABJQ7X6OGIUVNDBIZ4RNUKEEI3AEA, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:25.353897443 (3m41.134830808s) [noise=true , address=77.90.31.18:4805 , piece_id=DLOTLJTQIB7MPNFFX6ZJZKYM2SH6DCK42QQ6JSX44OP5X6F6FR2Q, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:25.406007691 (3m41.186941069s) [noise=true , address=77.90.31.10:4520 , piece_id=WVDWEFRRKQHEKOTCLLXLRFBOKNIIYP5A7UUG65D3DNDBC64BKKTQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: <nil> Jul 26 21:55:50.132368988 (4m5.913302372s) [noise=true , address=104.36.82.16:4294 , piece_id=LJC6J2ZGRGQCQCJTVRYKBITAEQDD4SWDVUXDIKH2ZJ6GHONHC4TQ, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:50.132475169 (4m5.913408545s) [noise=true , address=104.36.82.4:4803 , piece_id=QTE2FUEJ4SUC3YBZH22CP26N7ABI7XLJUNJ4ZVLH7V2YUXZWJ23Q, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:50.132612953 (4m5.913546337s) [noise=true , address=104.36.82.7:4909 , piece_id=MH34BK5ZDXUZX5XUUFR6YWOMRD6CQB2UHPN7RS7DWCBGVLISXVIA, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:55:50.132844023 (4m5.913777387s) [noise=true , address=104.36.82.6:4875 , piece_id=MV5GXWFIFYAZIOXE7CW7XP3QBCFZSF3AJXOGKK5GVYPUS3FFJJVA, seg_pos={0 0}, upload=50]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:55:50.132873238 (4m5.913806624s) [seg_pos={0 0}, upload=50]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:55:50.156981397 (4m5.937914767s) [upload=50]: Done uploading segments... Jul 26 21:55:50.157007415 (4m5.93794079s) [upload=50]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.027325927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 11.41616699s FAIL (2025-07-26 21:55:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 11.416203535s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.030155623s FAIL (2025-07-26 21:55:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 40:4508 , piece_id=ECURLKIEBB7MMUF2CCDFG5SOCHO25VK7JHJMTGSGMGMJQ3XSL7KQ, seg_pos={0 0}, upload=36]: Uploading piece... Jul 26 21:54:30.723752622 (2m46.504686005s) [noise=true , address=104.36.82.36:4631 , piece_id=UHLCEBXNZDGAQJQTCHDQ77B54A6HWAZZQC4MUFLJA7YIDEQWBEEA, seg_pos={0 0}, upload=36]: Uploading piece... Jul 26 21:54:30.724073807 (2m46.505007182s) [noise=true , address=104.36.82.19:4422 , piece_id=6OL2YACOBZWT6VYHMZHL46C5ZAQV2352MGPYKBCOEV6F46KLZYVQ, seg_pos={0 0}, upload=36]: Uploading piece... Jul 26 21:54:30.724132570 (2m46.505065953s) [noise=true , address=104.36.82.17:4351 , piece_id=6FRYTZO537CLDHDM3L3DXWDB5HYOAOONMGIWHALPUGA6BWREPCSQ, seg_pos={0 0}, upload=36]: Uploading piece... Jul 26 21:54:30.724690814 (2m46.505624199s) [noise=true , address=79.127.219.38:4006 , piece_id=Y6IEBRTPD7PLKIOQNG4SGQW2R337BWDSFI4OF7KM5AHFNCTWATMA, seg_pos={0 0}, upload=36]: Uploading piece... Jul 26 21:54:30.724726228 (2m46.505659613s) [noise=true , address=104.36.82.46:4505 , piece_id=JA7VIDE2DJFQMEUQZZTZY3G3SCYG7AFEEAM4OMLASC6XZH5WPM2Q, seg_pos={0 0}, upload=36]: Uploading piece... Jul 26 21:54:30.725118565 (2m46.506051944s) [noise=true , address=104.36.82.2:4758 , piece_id=UTJT7HXWNKF6TTPCJPJQMPHICF2D7E5HM7MTMR6JJ7QFWDZS4HMQ, seg_pos={0 0}, upload=36]: Uploading piece... Jul 26 21:54:30.725212483 (2m46.506145864s) [noise=true , address=104.36.82.56:4263 , piece_id=SQ3VYN6TMDBFYNAVMPN54XEOJ6QBBBZ3PEWKAQSQCZWWJD6MCZ2A, seg_pos={0 0}, upload=36]: Uploading piece... Jul 26 21:54:30.725724521 (2m46.506657901s) [noise=true , address=104.36.82.13:4217 , piece_id=6V7KT3HRW2AKR7BREQLCNGWEAMVLTJCSXATTKNHANJVVN2SA3L5Q, seg_pos={0 0}, upload=36]: Uploading piece... Jul 26 21:54:30.789103932 (2m46.570037318s) [upload=36]: Next returned nil segment Jul 26 21:54:30.789711742 (2m46.570645127s) [upload=36]: Waiting for error group managing segments... Jul 26 21:54:31.287065550 (2m47.067998929s) [noise=true , address=79.127.219.38:4006 , piece_id=Y6IEBRTPD7PLKIOQNG4SGQW2R337BWDSFI4OF7KM5AHFNCTWATMA, seg_pos={0 0}, upload=36]: Done uploading piece. err: <nil> Jul 26 21:54:33.089818798 (2m48.870752179s) [noise=true , address=79.127.201.215:4345 , piece_id=CDZVXSNU4ZWKQMERE76AX4ZRY3DKZVGQPVBNYLMU7JSVUYOVPZGQ, seg_pos={0 0}, upload=36]: Done uploading piece. err: <nil> Jul 26 21:54:36.516525271 (2m52.297458656s) [noise=true , address=64.49.13.140:4508 , piece_id=ECURLKIEBB7MMUF2CCDFG5SOCHO25VK7JHJMTGSGMGMJQ3XSL7KQ, seg_pos={0 0}, upload=36]: Done uploading piece. err: <nil> Jul 26 21:55:30.651507682 (3m46.432441066s) [noise=true , address=104.36.82.52:4090 , piece_id=3XKMXVH2XLFOLLQU7VHNBHDZYUTWE6EW7GRHZAHJYODSVWZHX6OQ, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: write tcp 159.223.143.39:40350->104.36.82.52:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.651596468 (3m46.432529851s) [noise=true , address=104.36.82.4:4821 , piece_id=LZXSCFW3IZYIJABB2XAMBN562KYPGFKX4P54KVGU72K2FKTZ2UCQ, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:12TwpADh23qQ4v6WajK7wdaDrrTbWrQxRg6oQA48rZBDSxCK9rE, address:104.36.82.4:4821): protocol: write tcp 159.223.143.39:32984->104.36.82.4:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.651661099 (3m46.43259448s) [noise=true , address=104.36.82.2:4758 , piece_id=UTJT7HXWNKF6TTPCJPJQMPHICF2D7E5HM7MTMR6JJ7QFWDZS4HMQ, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:1joBbXdmusUXDfe1ofLp5oXy1nVkkRAng8QGL2hAtKjDz71u5g, address:104.36.82.2:4758): protocol: write tcp 159.223.143.39:44048->104.36.82.2:4758: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.651728392 (3m46.432661773s) [noise=true , address=104.36.82.56:4263 , piece_id=SQ3VYN6TMDBFYNAVMPN54XEOJ6QBBBZ3PEWKAQSQCZWWJD6MCZ2A, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:12tLW4r8AQKLCbSWZoNUVi5v2YtF8NthUmX81TLQn8G8TotuByG, address:104.36.82.56:4263): protocol: write tcp 159.223.143.39:33834->104.36.82.56:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652017205 (3m46.432950585s) [noise=true , address=104.36.82.54:4171 , piece_id=4HMVPOPT7IHXNMZN5JN524PXZZKK7INK3YYQKH2EFZT34AFYODXQ, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: write tcp 159.223.143.39:34332->104.36.82.54:4171: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652090241 (3m46.433023622s) [noise=true , address=104.36.82.38:4696 , piece_id=4G5UVH23DXTOMZ4K6MSKO2YA36W6XLRRZ356R7B6KBLKDFDV3JBA, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:1JrFvHgEHVLtwK3QYvFTzASCysBCR38qA4Ni8Jvrb1rXUaFy3H, address:104.36.82.38:4696): protocol: write tcp 159.223.143.39:45202->104.36.82.38:4696: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652269500 (3m46.433202888s) [noise=true , address=104.36.82.36:4631 , piece_id=UHLCEBXNZDGAQJQTCHDQ77B54A6HWAZZQC4MUFLJA7YIDEQWBEEA, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y, address:104.36.82.36:4631): protocol: write tcp 159.223.143.39:55484->104.36.82.36:4631: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652344017 (3m46.433277399s) [noise=true , address=104.36.82.17:4351 , piece_id=6FRYTZO537CLDHDM3L3DXWDB5HYOAOONMGIWHALPUGA6BWREPCSQ, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:17bJjsLQc1F8ARek42WZXsJgNPtbzGn3Badv7X9UhD8auyBP3j, address:104.36.82.17:4351): protocol: write tcp 159.223.143.39:41866->104.36.82.17:4351: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652414911 (3m46.433348295s) [noise=true , address=104.36.82.47:4529 , piece_id=7MSEPRQUHOX4BKCKWOCQBZ52BBLSYX5CTAGVKE3VVYN6WJLM22SQ, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:1m7FrQEps7d8enRYk4v17UxcDs8o8L2CFcVEyB3zR4EniXrYeH, address:104.36.82.47:4529): protocol: write tcp 159.223.143.39:40200->104.36.82.47:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652480788 (3m46.433414172s) [noise=true , address=104.36.82.46:4505 , piece_id=JA7VIDE2DJFQMEUQZZTZY3G3SCYG7AFEEAM4OMLASC6XZH5WPM2Q, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf, address:104.36.82.46:4505): protocol: write tcp 159.223.143.39:33492->104.36.82.46:4505: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652565390 (3m46.43349877s) [noise=true , address=104.36.82.13:4217 , piece_id=6V7KT3HRW2AKR7BREQLCNGWEAMVLTJCSXATTKNHANJVVN2SA3L5Q, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:123KLtoTMCQ9DZUjJ8JJab4D5bc7UGTt7yjVV8wDgFTozdZ5gpi, address:104.36.82.13:4217): protocol: write tcp 159.223.143.39:60152->104.36.82.13:4217: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652570468 (3m46.433503837s) [noise=true , address=104.36.82.40:4945 , piece_id=4OB4J3ON4B3OZNZ6A656ZRRV46IF6FZBBXLUOS3QASIX7GGU3VYA, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:104.36.82.40:4945): protocol: write tcp 159.223.143.39:42586->104.36.82.40:4945: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652627403 (3m46.433560781s) [noise=true , address=104.36.82.55:4216 , piece_id=UJT6HUWDPLFSAPWBAVEQIH6TVRQJWXA3DRUYFALMRI6KZKD44JAA, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: write tcp 159.223.143.39:46480->104.36.82.55:4216: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652640188 (3m46.433573566s) [noise=true , address=104.36.82.22:4539 , piece_id=PQN7S3OYDWYLHKQFW43KAFZEFPEVIPRZKI66MG47GF7QPJUNBMYQ, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:1QL6w8DSFR6CxDdnJmDuyVZxp6JU55VKiJofBZRfQi7q31i8Jc, address:104.36.82.22:4539): protocol: write tcp 159.223.143.39:54686->104.36.82.22:4539: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652686771 (3m46.433620148s) [noise=true , address=104.36.82.19:4422 , piece_id=6OL2YACOBZWT6VYHMZHL46C5ZAQV2352MGPYKBCOEV6F46KLZYVQ, seg_pos={0 0}, upload=36]: Done uploading piece. err: ecclient: upload failed (node:1k7Bs5nYevGzsVWhp85cgDsbrpzbefborGwMAAnDsbeT9x7gio, address:104.36.82.19:4422): protocol: write tcp 159.223.143.39:39382->104.36.82.19:4422: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:30.652735871 (3m46.433669255s) [seg_pos={0 0}, upload=36]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 26 21:55:30.677869322 (3m46.458802715s) [upload=36]: Done uploading segments... Jul 26 21:55:30.677904947 (3m46.458838329s) [upload=36]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.030234793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.030657526s FAIL (2025-07-26 21:55:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== upload=31]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 159.223.143.39:58086->77.90.31.18:4805: use of closed network connection; write tcp 159.223.143.39:58086->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58086->77.90.31.18:4805: use of closed network connection Jul 26 21:54:12.123344461 (2m27.904277848s) [seg_pos={0 0}, upload=31]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:54:12.123745142 (2m27.904678776s) [upload=31]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:54:12.123762481 (2m27.90469586s) [upload=31]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:54:12.169238805 (2m27.950172175s) [upload=31]: Done uploading segments... Jul 26 21:54:12.169257773 (2m27.95019115s) [upload=31]: Upload finished. err: <nil> Jul 26 21:54:50.073620192 (3m5.85455357s) [upload=43]: Starting upload Jul 26 21:54:50.073715857 (3m5.854649235s) [upload=43]: Done starting upload Jul 26 21:54:50.073800386 (3m5.854733763s) [upload=43]: Uploading segments... Jul 26 21:54:50.074069548 (3m5.855002926s) [upload=43]: Got next segment. Inline: false Jul 26 21:54:50.074132303 (3m5.855065681s) [upload=43]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:54:50.074139261 (3m5.855072639s) [upload=43]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:54:50.107223526 (3m5.888156913s) [seg_pos={0 0}, upload=43]: Begin upload segment... Jul 26 21:54:50.107279038 (3m5.888212418s) [seg_pos={0 0}, upload=43]: Done begin upload segment. Jul 26 21:54:50.107312843 (3m5.88824623s) [noise=true , address=104.36.82.7:4912 , piece_id=5MIGSQ4NK3ZCCNMYAB6WEYQ36BWZDRKPZXZBGFEE6P3EEDAL6ZBQ, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.107326680 (3m5.88826006s) [noise=true , address=45.91.116.9:4263 , piece_id=FIMCCT5FKMRVHRVXXDM7U4NHB353BWEW6GSTOJ6UYNJ7MYNVZOBA, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.107650165 (3m5.888583558s) [noise=true , address=45.91.116.10:4082 , piece_id=HFREVTENXTAIMTNPJQXT3CLV27VXNZFSRXXSQGRS6JBH5YOAG5ZA, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.107816614 (3m5.88875s ) [noise=true , address=45.91.116.4:4939 , piece_id=5HV2AB3Q7RTRENBOMU37BUM7PZUADWHJNVCTXSNSYFXT7ZVUF7NA, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.108344030 (3m5.889277413s) [noise=true , address=77.90.31.21:4877 , piece_id=PC2NLWFOXKYS6J4JOO6DTBVKCNH27U5RAY7R7WUBD3BLRX5P4M4Q, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.108415099 (3m5.889348483s) [noise=true , address=77.90.31.11:4553 , piece_id=7AJQ4YTCAQLPBOGZSKUJZRGUJEF7VICDEFLULYC6QXPS3DBPNRKQ, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.108832012 (3m5.889765402s) [noise=true , address=104.36.82.14:4223 , piece_id=4F6ADV22WOQSN3HKFM4JCVZA3OWSYX2S6OP2TAH4SBOKHJ2AQCYQ, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.108995554 (3m5.889928942s) [noise=true , address=45.91.116.8:4045 , piece_id=BKDSFORXBPR2J4MOMZGN4BIF52TLZKAZVNN5NFX5E6SIFNG4A3KA, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.109384982 (3m5.890318373s) [noise=true , address=45.91.116.14:4191 , piece_id=EEO6QITUY3RAGRBIZ4UEMVMN4FRGE3RHNTJNHZDCGC37FECP7QQA, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.109424667 (3m5.890358051s) [noise=true , address=45.91.116.10:4083 , piece_id=JNGBOCN7EMTP4R756WMOEHUCRG7VO4CSSGAZCVHB5D23BFETNKCA, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.109857552 (3m5.890790939s) [noise=true , address=104.36.82.8:4969 , piece_id=OECWQP6UP6YVE6IYZ32ZRLXG2ZDRX4HQJNZ4LA2HGUYEMLC4NHUQ, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.109892222 (3m5.89082559s) [noise=true , address=45.91.116.12:4226 , piece_id=S6BU5LUOYWALS3ZHCZ7YOGDK22NOMSU3HSKOJJYNC74UKZM2SCEA, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.110281520 (3m5.891214904s) [noise=true , address=104.36.82.11:4115 , piece_id=HUFPWTAH5OZ52WABBCWRUJYUVIIJUDVYW7Y3VFWBHJM63RUX5MZA, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.110734661 (3m5.89166805s) [noise=true , address=45.91.116.11:4117 , piece_id=LNOZYIR3KYCJF6EOHMVWES4BHG25DWUGKTRIAL7DDUWH4ROZRO5Q, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.111232631 (3m5.892166016s) [noise=true , address=45.91.116.13:4154 , piece_id=RVLCLDTNFJ6JH3GQNOFCF5VCCN3E733V7D6W7KG6VPQDAH7YH4SA, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.112232954 (3m5.89316634s) [noise=true , address=104.36.82.13:4186 , piece_id=3P4TVOZ2BRS2RJT2C6HWXK2ZDN5ZCHAQPWXSMPOD63653PUAZOOA, seg_pos={0 0}, upload=43]: Uploading piece... Jul 26 21:54:50.220496506 (3m6.0014299s) [upload=43]: Next returned nil segment Jul 26 21:54:50.220518704 (3m6.001452091s) [upload=43]: Waiting for error group managing segments... Jul 26 21:54:50.224864247 (3m6.005797618s) [noise=true , address=45.91.116.13:4154 , piece_id=RVLCLDTNFJ6JH3GQNOFCF5VCCN3E733V7D6W7KG6VPQDAH7YH4SA, seg_pos={0 0}, upload=43]: Done uploading piece. err: <nil> Jul 26 21:54:50.226189598 (3m6.007122979s) [noise=true , address=45.91.116.8:4045 , piece_id=BKDSFORXBPR2J4MOMZGN4BIF52TLZKAZVNN5NFX5E6SIFNG4A3KA, seg_pos={0 0}, upload=43]: Done uploading piece. err: <nil> Jul 26 21:54:50.227291515 (3m6.008224895s) [noise=true , address=45.91.116.10:4083 , piece_id=JNGBOCN7EMTP4R756WMOEHUCRG7VO4CSSGAZCVHB5D23BFETNKCA, seg_pos={0 0}, upload=43]: Done uploading piece. err: <nil> Jul 26 21:54:50.228053790 (3m6.008987169s) [noise=true , address=45.91.116.10:4082 , piece_id=HFREVTENXTAIMTNPJQXT3CLV27VXNZFSRXXSQGRS6JBH5YOAG5ZA, seg_pos={0 0}, upload=43]: Done uploading piece. err: <nil> Jul 26 21:54:50.228885243 (3m6.009818621s) [noise=true , address=45.91.116.12:4226 , piece_id=S6BU5LUOYWALS3ZHCZ7YOGDK22NOMSU3HSKOJJYNC74UKZM2SCEA, seg_pos={0 0}, upload=43]: Done uploading piece. err: <nil> Jul 26 21:54:50.228908606 (3m6.009841975s) [noise=true , address=45.91.116.4:4939 , piece_id=5HV2AB3Q7RTRENBOMU37BUM7PZUADWHJNVCTXSNSYFXT7ZVUF7NA, seg_pos={0 0}, upload=43]: Done uploading piece. err: <nil> Jul 26 21:54:50.229514878 (3m6.010448256s) [noise=true , address=45.91.116.11:4117 , piece_id=LNOZYIR3KYCJF6EOHMVWES4BHG25DWUGKTRIAL7DDUWH4ROZRO5Q, seg_pos={0 0}, upload=43]: Done uploading piece. err: <nil> Jul 26 21:54:50.229598513 (3m6.010531897s) [noise=true , address=45.91.116.14:4191 , piece_id=EEO6QITUY3RAGRBIZ4UEMVMN4FRGE3RHNTJNHZDCGC37FECP7QQA, seg_pos={0 0}, upload=43]: Done uploading piece. err: <nil> Jul 26 21:54:50.230258515 (3m6.011191877s) [noise=true , address=45.91.116.9:4263 , piece_id=FIMCCT5FKMRVHRVXXDM7U4NHB353BWEW6GSTOJ6UYNJ7MYNVZOBA, seg_pos={0 0}, upload=43]: Done uploading piece. err: <nil> Jul 26 21:54:58.478593321 (3m14.259526699s) [noise=true , address=77.90.31.21:4877 , piece_id=PC2NLWFOXKYS6J4JOO6DTBVKCNH27U5RAY7R7WUBD3BLRX5P4M4Q, seg_pos={0 0}, upload=43]: Done uploading piece. err: <nil> Jul 26 21:54:58.911319908 (3m14.692253273s) [noise=true , address=77.90.31.11:4553 , piece_id=7AJQ4YTCAQLPBOGZSKUJZRGUJEF7VICDEFLULYC6QXPS3DBPNRKQ, seg_pos={0 0}, upload=43]: Done uploading piece. err: <nil> Jul 26 21:55:20.037193534 (3m35.818126925s) [noise=true , address=104.36.82.8:4969 , piece_id=OECWQP6UP6YVE6IYZ32ZRLXG2ZDRX4HQJNZ4LA2HGUYEMLC4NHUQ, seg_pos={0 0}, upload=43]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:46722->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037216475 (3m35.818149858s) [noise=true , address=104.36.82.11:4115 , piece_id=HUFPWTAH5OZ52WABBCWRUJYUVIIJUDVYW7Y3VFWBHJM63RUX5MZA, seg_pos={0 0}, upload=43]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:47184->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037313465 (3m35.818246864s) [noise=true , address=104.36.82.7:4912 , piece_id=5MIGSQ4NK3ZCCNMYAB6WEYQ36BWZDRKPZXZBGFEE6P3EEDAL6ZBQ, seg_pos={0 0}, upload=43]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037373424 (3m35.818306814s) [noise=true , address=104.36.82.13:4186 , piece_id=3P4TVOZ2BRS2RJT2C6HWXK2ZDN5ZCHAQPWXSMPOD63653PUAZOOA, seg_pos={0 0}, upload=43]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037396511 (3m35.818329895s) [noise=true , address=104.36.82.14:4223 , piece_id=4F6ADV22WOQSN3HKFM4JCVZA3OWSYX2S6OP2TAH4SBOKHJ2AQCYQ, seg_pos={0 0}, upload=43]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:51250->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:20.037425149 (3m35.818358546s) [seg_pos={0 0}, upload=43]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 21:55:20.065111780 (3m35.846045157s) [upload=43]: Done uploading segments... Jul 26 21:55:20.065145366 (3m35.846078742s) [upload=43]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.030692379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 4.291123783s FAIL (2025-07-26 21:54:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 4.291146906s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.048254659s FAIL (2025-07-26 21:54:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.048282697s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.032386407s FAIL (2025-07-26 21:54:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== noise=true , address=104.36.82.53:4152 , piece_id=DNY53NPNRZANEBJ6FUUNW6CFTFM3HPNCB7CKIT6IVSG4PJX6XKTA, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.640955065 (1m46.421888448s) [noise=true , address=79.127.201.215:4360 , piece_id=Q3HETDFKTKEYFBDV2KZJZH52IPQS5I5CHOI3X6G42BBX5J7TXAIA, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.641389206 (1m46.422322592s) [noise=true , address=104.36.82.51:4065 , piece_id=REEYCSOZ6AZJS3765A33BFGL5YFGOEMVPMFFRYUHJ44WYLWM3FEQ, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.641407977 (1m46.422341364s) [noise=true , address=104.36.82.16:4322 , piece_id=33GL35JXQYY7URZBBFYBD2KLOBWUOMW446R2QS5BHB7UPV6R7TXA, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.641842121 (1m46.422775511s) [noise=true , address=104.36.82.34:4573 , piece_id=7Z62FTQTYVNIIYDMVKAJVCNZ34CBDTB65AS76AFTNBMZILKZ75TA, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.641972912 (1m46.422906301s) [noise=true , address=104.36.82.39:4740 , piece_id=J3USUN6GYMJDFCJFLUORJPPRPBEMGCQC2OAPDL5GMWX6MRHXPNEA, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.642296088 (1m46.423229476s) [noise=true , address=64.49.13.140:4515 , piece_id=2SNA4RTTS4FR755DHRLULMQRYQAMCMK3KREU34DERTJ6OQ4ZO33Q, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.642484548 (1m46.423417937s) [noise=true , address=104.36.82.46:4497 , piece_id=SS2CAJV7UX6OEKURDF4GNP6GHVD7JHRCLF77POXFNIJHRRADHUEQ, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.642730005 (1m46.423663395s) [noise=true , address=64.49.13.170:4718 , piece_id=HVTHF2RIR4XSB5XIACS7FQF5JPTQ4F5DUGCEWYWCYBPR7N7V5CXQ, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.642999916 (1m46.423933352s) [noise=true , address=45.91.116.14:4206 , piece_id=WNWIR5YYONMDFE5ETWBTF3U7AH4OPL4IPIY3CY44IXRB6XUJKTHA, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.643206334 (1m46.424139721s) [noise=true , address=77.90.31.15:4717 , piece_id=6MARY3MZF6PKP4KHKOBPWQJZT5AFEGMU7ZKFF4SBDB4JFYLHBXNA, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.643447298 (1m46.424380686s) [noise=true , address=104.36.82.56:4262 , piece_id=DKJDIFM2X2UFA36PEJ7N54AEQUS2OTNT55MITYKGCF3RFAHYDE6Q, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.643626180 (1m46.42455957s) [noise=true , address=104.36.82.55:4220 , piece_id=6EQSENNTM4VG7TOQDI2PHAWL6ILT4LNVOPHLXAOU4VPYIQ7HMARA, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.644055884 (1m46.42498927s) [noise=true , address=104.36.82.5:4860 , piece_id=XQOJHEJZ6UGNF776TE5XJJCQ7IZYAHBZY2QIOA7QRBBM6C52NZQQ, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.644572540 (1m46.425505927s) [noise=true , address=79.127.219.38:4010 , piece_id=I7BL7BEMJXQC7X7ZINQMCRFVMLN3DCLJWJNEZGXVD6GMOZXDFD3Q, seg_pos={0 0}, upload=17]: Uploading piece... Jul 26 21:53:30.717908055 (1m46.498841439s) [upload=17]: Next returned nil segment Jul 26 21:53:30.718420191 (1m46.499353579s) [upload=17]: Waiting for error group managing segments... Jul 26 21:53:30.723935721 (1m46.504869092s) [noise=true , address=79.127.219.38:4010 , piece_id=I7BL7BEMJXQC7X7ZINQMCRFVMLN3DCLJWJNEZGXVD6GMOZXDFD3Q, seg_pos={0 0}, upload=17]: Done uploading piece. err: <nil> Jul 26 21:53:30.821918847 (1m46.60285223s) [noise=true , address=45.91.116.14:4206 , piece_id=WNWIR5YYONMDFE5ETWBTF3U7AH4OPL4IPIY3CY44IXRB6XUJKTHA, seg_pos={0 0}, upload=17]: Done uploading piece. err: <nil> Jul 26 21:53:31.513503290 (1m47.294436657s) [noise=true , address=64.49.13.140:4515 , piece_id=2SNA4RTTS4FR755DHRLULMQRYQAMCMK3KREU34DERTJ6OQ4ZO33Q, seg_pos={0 0}, upload=17]: Done uploading piece. err: <nil> Jul 26 21:53:31.534302793 (1m47.315236177s) [noise=true , address=64.49.13.170:4718 , piece_id=HVTHF2RIR4XSB5XIACS7FQF5JPTQ4F5DUGCEWYWCYBPR7N7V5CXQ, seg_pos={0 0}, upload=17]: Done uploading piece. err: <nil> Jul 26 21:53:35.230487822 (1m51.011421205s) [noise=true , address=77.90.31.15:4717 , piece_id=6MARY3MZF6PKP4KHKOBPWQJZT5AFEGMU7ZKFF4SBDB4JFYLHBXNA, seg_pos={0 0}, upload=17]: Done uploading piece. err: <nil> Jul 26 21:53:42.848504874 (1m58.629438241s) [noise=true , address=79.127.201.215:4360 , piece_id=Q3HETDFKTKEYFBDV2KZJZH52IPQS5I5CHOI3X6G42BBX5J7TXAIA, seg_pos={0 0}, upload=17]: Done uploading piece. err: <nil> Jul 26 21:54:30.555111196 (2m46.33604459s) [noise=true , address=104.36.82.16:4322 , piece_id=33GL35JXQYY7URZBBFYBD2KLOBWUOMW446R2QS5BHB7UPV6R7TXA, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:104.36.82.16:4322): protocol: rpc: dial tcp 104.36.82.16:4322: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555212728 (2m46.336146115s) [noise=true , address=104.36.82.53:4152 , piece_id=DNY53NPNRZANEBJ6FUUNW6CFTFM3HPNCB7CKIT6IVSG4PJX6XKTA, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: rpc: dial tcp 104.36.82.53:4152: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555278048 (2m46.336211427s) [noise=true , address=104.36.82.52:4103 , piece_id=QMCP7XYULICAGKFW5CFAESQXMZ3ZQJRHYVRQT4L4PDIRLIRQNB6Q, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: rpc: dial tcp 104.36.82.52:4103: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555339447 (2m46.33627285s) [noise=true , address=104.36.82.39:4740 , piece_id=J3USUN6GYMJDFCJFLUORJPPRPBEMGCQC2OAPDL5GMWX6MRHXPNEA, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:1A2tCUpDAzrSzhUfBf6GMsZdpY6SPkRHWnqrWiiy7uNGfzcYbq, address:104.36.82.39:4740): protocol: rpc: dial tcp 104.36.82.39:4740: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555409749 (2m46.336343128s) [noise=true , address=104.36.82.55:4220 , piece_id=6EQSENNTM4VG7TOQDI2PHAWL6ILT4LNVOPHLXAOU4VPYIQ7HMARA, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: rpc: dial tcp 104.36.82.55:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555475618 (2m46.336409006s) [noise=true , address=104.36.82.5:4860 , piece_id=XQOJHEJZ6UGNF776TE5XJJCQ7IZYAHBZY2QIOA7QRBBM6C52NZQQ, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:1Qr9fghoRVbAvNm9AMEHhhtc7JqHWBuZerBttg3KW3J3GcDjrE, address:104.36.82.5:4860): protocol: rpc: dial tcp 104.36.82.5:4860: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555553857 (2m46.336487242s) [noise=true , address=104.36.82.51:4065 , piece_id=REEYCSOZ6AZJS3765A33BFGL5YFGOEMVPMFFRYUHJ44WYLWM3FEQ, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp, address:104.36.82.51:4065): protocol: rpc: dial tcp 104.36.82.51:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555643939 (2m46.336577318s) [noise=true , address=104.36.82.54:4174 , piece_id=JSHRXNQ3IQAABDKTJHLJVAOZJYLOOV3JULB6PS5YAWLILPIFETAA, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: rpc: dial tcp 104.36.82.54:4174: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555696058 (2m46.336629443s) [noise=true , address=104.36.82.47:4538 , piece_id=7SDIGILXC6Y7DVCGKNEBXEEXV7AX7CN25UVQZUGTMFL2U5CLTHWQ, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555733975 (2m46.336667367s) [noise=true , address=104.36.82.34:4573 , piece_id=7Z62FTQTYVNIIYDMVKAJVCNZ34CBDTB65AS76AFTNBMZILKZ75TA, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:1yD3c1tEtGt2yYEyzDX5GugUChtkCQa9VTWkBH9ftgwvqn72SU, address:104.36.82.34:4573): protocol: rpc: dial tcp 104.36.82.34:4573: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555759150 (2m46.33669253s) [noise=true , address=104.36.82.56:4262 , piece_id=DKJDIFM2X2UFA36PEJ7N54AEQUS2OTNT55MITYKGCF3RFAHYDE6Q, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:1DY2kpdCo8EDSs1LHL6qBFo6qjnsb7FL1QDekMV5qaWB5jxeMz, address:104.36.82.56:4262): protocol: rpc: dial tcp 104.36.82.56:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555789867 (2m46.336723245s) [noise=true , address=104.36.82.46:4497 , piece_id=SS2CAJV7UX6OEKURDF4GNP6GHVD7JHRCLF77POXFNIJHRRADHUEQ, seg_pos={0 0}, upload=17]: Done uploading piece. err: ecclient: upload failed (node:12Zg4PSYBuLxFvF9oYiuvn65JNbjT5RdagcVAQLp2izvjPe1NEc, address:104.36.82.46:4497): protocol: write tcp 159.223.143.39:44442->104.36.82.46:4497: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:54:30.555851943 (2m46.336785325s) [seg_pos={0 0}, upload=17]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 21:54:30.585037970 (2m46.365971343s) [upload=17]: Done uploading segments... Jul 26 21:54:30.585088541 (2m46.366021921s) [upload=17]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.032407821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.033138482s FAIL (2025-07-26 21:53:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 1:52:30.530832740 (46.311766131s) [noise=true , address=79.127.219.38:4010 , piece_id=TKO42BNX2MCVL4DZC3HMGC3RDA3LXJHDL4C53ZWTYXF3ZPKFTW5A, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.531296581 (46.312229969s) [noise=true , address=104.36.82.47:4528 , piece_id=YGSYWRKXDJ4M5MRUKP4JUUBPYJ6YVHFZ2M6Y4EAW7Q6FMPZN5SQQ, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.531685529 (46.312618915s) [noise=true , address=79.127.201.215:4353 , piece_id=MKWGU5DWN3UHNLRMV4J3ANG3IPD5I3WMP6U6VYWOVEQ7QQT2ZH2A, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.532097220 (46.313030608s) [noise=true , address=104.36.82.49:4999 , piece_id=ACES3OKT2W4CYGVL5K2UQ7LL3IHGVDU4DYTOEW5DM2ISI4FRFFFA, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.532854731 (46.313788118s) [noise=true , address=104.36.82.55:4218 , piece_id=HJDUK2UIAEAVMEHEZ42X6D337BMU3RY44L5T6NK3L46EYQSF4RVA, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.533393833 (46.314327223s) [noise=true , address=104.36.82.37:4676 , piece_id=6JEHEUESDVEVJNQWECNLWPVKNE3UVYIS2YS6RZWLZPXTYZQG64KQ, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.533901272 (46.314834658s) [noise=true , address=104.36.82.16:4299 , piece_id=AJREN52SBTKGAPFP52OT2QSM6P5DBALYAAX36N5IFVSN6DIUAD7A, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.534393515 (46.31532691s) [noise=true , address=104.36.82.3:4784 , piece_id=ZBEUFD6JKNSZFK4CNHMZHXLA7MAK7RHLOPGUV7PRWQ7DDXOJMMAQ, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.534892174 (46.315825558s) [noise=true , address=104.36.82.53:4131 , piece_id=VXYHRP2O2HO5I6GPLGWCUTFED5T3M6BAFKTUCSE5N6VLYHMG6ZCQ, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.536439593 (46.317372981s) [noise=true , address=104.36.82.52:4085 , piece_id=LK5EBDP7OMBNXFKBWQXB653PTXBJUNSJK3HON2PSIJ3BOLXY6SMA, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.537001596 (46.317934983s) [noise=true , address=79.127.205.235:4688 , piece_id=JDW6LATWMQVNYZEFKIZL5QNOFGWCIJLJJIW23NVH6S5DB4PUKTHA, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.537490027 (46.318423416s) [noise=true , address=104.36.82.3:4771 , piece_id=2LPKYHCZ7EA3LUABFV4KWNS7UJZZKIZGOISRSZJAVGNCTJENZUSA, seg_pos={0 0}, upload=3]: Uploading piece... Jul 26 21:52:30.707817486 (46.488750865s) [upload=3]: Next returned nil segment Jul 26 21:52:30.707839698 (46.488773086s) [upload=3]: Waiting for error group managing segments... Jul 26 21:52:30.828067348 (46.609000738s) [noise=true , address=79.127.219.38:4010 , piece_id=TKO42BNX2MCVL4DZC3HMGC3RDA3LXJHDL4C53ZWTYXF3ZPKFTW5A, seg_pos={0 0}, upload=3]: Done uploading piece. err: <nil> Jul 26 21:52:31.239432163 (47.020365562s) [noise=true , address=79.127.205.235:4688 , piece_id=JDW6LATWMQVNYZEFKIZL5QNOFGWCIJLJJIW23NVH6S5DB4PUKTHA, seg_pos={0 0}, upload=3]: Done uploading piece. err: <nil> Jul 26 21:52:44.375272085 (1m0.156205455s) [noise=true , address=79.127.201.215:4353 , piece_id=MKWGU5DWN3UHNLRMV4J3ANG3IPD5I3WMP6U6VYWOVEQ7QQT2ZH2A, seg_pos={0 0}, upload=3]: Done uploading piece. err: <nil> Jul 26 21:53:30.442852617 (1m46.223786003s) [noise=true , address=104.36.82.17:4349 , piece_id=4WRC2JZZP4TWMC42QKSABBU5W3KFDUJMEMXV2OTLBLHYUQRVNMSQ, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:1zywsXJnw5gfpUu7mC8p7UE7ev5HexPu5nSpUH1x2pzar99T97, address:104.36.82.17:4349): protocol: rpc: dial tcp 104.36.82.17:4349: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.442951652 (1m46.223885038s) [noise=true , address=104.36.82.49:4999 , piece_id=ACES3OKT2W4CYGVL5K2UQ7LL3IHGVDU4DYTOEW5DM2ISI4FRFFFA, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:1tSiyPFy1xn1YRCLmKmPmbTm3LUmDP9nT1YRCXnsuyY6XsLKrG, address:104.36.82.49:4999): protocol: rpc: dial tcp 104.36.82.49:4999: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.443015988 (1m46.223949353s) [noise=true , address=104.36.82.37:4676 , piece_id=6JEHEUESDVEVJNQWECNLWPVKNE3UVYIS2YS6RZWLZPXTYZQG64KQ, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:18yqKh7gMSrXWavfChEJRX7tV5rS9xTSo4SbmgPMtnYKfgbvF8, address:104.36.82.37:4676): protocol: rpc: dial tcp 104.36.82.37:4676: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.443131008 (1m46.224064391s) [noise=true , address=104.36.82.46:4484 , piece_id=4DJPSJV4UP4W4KMDKTC4OHKYEFYRMEUXYXXN3SAO3FKFPYIYHOLA, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: rpc: dial tcp 104.36.82.46:4484: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.443224053 (1m46.224157504s) [noise=true , address=104.36.82.45:4936 , piece_id=LO6FPCYTRG2COZLY6DMUCBMM4NVUUA2JGTTFWOOXCBAO6CKNIPFA, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8, address:104.36.82.45:4936): protocol: rpc: dial tcp 104.36.82.45:4936: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.443259332 (1m46.224192714s) [noise=true , address=104.36.82.3:4784 , piece_id=ZBEUFD6JKNSZFK4CNHMZHXLA7MAK7RHLOPGUV7PRWQ7DDXOJMMAQ, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:1534abg2Jn4zTQzvBcDDZDLq2QZp6kBKY1q6zEbk2PC8mJrnq3, address:104.36.82.3:4784): protocol: rpc: dial tcp 104.36.82.3:4784: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.443393358 (1m46.224326735s) [noise=true , address=104.36.82.56:4249 , piece_id=JCRASXLHIAZMUEIKWXHFOWAIWV2MRLVCCSHNOQEBK3JFJBXWVQ6Q, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: rpc: dial tcp 104.36.82.56:4249: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.443492380 (1m46.224425757s) [noise=true , address=104.36.82.51:4050 , piece_id=HGOND6DGGL3YAIRPKU2DOAQK4REXYKNONAR26CGOFLIMRTIXSO3A, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:104.36.82.51:4050): protocol: rpc: dial tcp 104.36.82.51:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.443631243 (1m46.224564624s) [noise=true , address=104.36.82.47:4528 , piece_id=YGSYWRKXDJ4M5MRUKP4JUUBPYJ6YVHFZ2M6Y4EAW7Q6FMPZN5SQQ, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: rpc: dial tcp 104.36.82.47:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.443746502 (1m46.224679886s) [noise=true , address=104.36.82.55:4218 , piece_id=HJDUK2UIAEAVMEHEZ42X6D337BMU3RY44L5T6NK3L46EYQSF4RVA, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: rpc: dial tcp 104.36.82.55:4218: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.443879086 (1m46.224812478s) [noise=true , address=104.36.82.53:4131 , piece_id=VXYHRP2O2HO5I6GPLGWCUTFED5T3M6BAFKTUCSE5N6VLYHMG6ZCQ, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:1uhyc5ozrF9ndeEnQXtjc9o3tsYiAnW1dzeuHV3kv1JS4T4a8C, address:104.36.82.53:4131): protocol: write tcp 159.223.143.39:33160->104.36.82.53:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.443982140 (1m46.224915517s) [noise=true , address=104.36.82.3:4771 , piece_id=2LPKYHCZ7EA3LUABFV4KWNS7UJZZKIZGOISRSZJAVGNCTJENZUSA, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:1rcgrMxg2fRfvh7EA1eUehzD5rSH1krUd5W5eC8KTo7vjYWFLL, address:104.36.82.3:4771): protocol: write tcp 159.223.143.39:47894->104.36.82.3:4771: use of closed network connection; context deadline exceeded Jul 26 21:53:30.443992288 (1m46.224925715s) [noise=true , address=104.36.82.11:4122 , piece_id=PMHPGDUAICVVUQSMVK2A4523HWKSARIAGWOIJPT2U3XY7Q66ZRKQ, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op Jul 26 21:53:30.444103408 (1m46.225036793s) [noise=true , address=104.36.82.16:4299 , piece_id=AJREN52SBTKGAPFP52OT2QSM6P5DBALYAAX36N5IFVSN6DIUAD7A, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n, address:104.36.82.16:4299): protocol: rpc: dial tcp 104.36.82.16:4299: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.444118128 (1m46.225051497s) [noise=true , address=104.36.82.52:4085 , piece_id=LK5EBDP7OMBNXFKBWQXB653PTXBJUNSJK3HON2PSIJ3BOLXY6SMA, seg_pos={0 0}, upload=3]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: rpc: dial tcp 104.36.82.52:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:53:30.444144145 (1m46.225077533s) [seg_pos={0 0}, upload=3]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 26 21:53:30.472496138 (1m46.253429523s) [upload=3]: Done uploading segments... Jul 26 21:53:30.472536099 (1m46.253469516s) [upload=3]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.033162972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................X.......X.......X...X......] duration: 5.231042094s FAIL (2025-07-26 21:51:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.23106995s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................X.XX...X.X...X.....] duration: 6.938693393s FAIL (2025-07-26 21:51:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 57.390018434 (58h10m40.234819956s) [noise=true , address=104.36.82.11:4115 , piece_id=OJKMPUDNWYKMQOUQEYRDZC2CLCSRDJ56K25VRIM6KKTMQGIY6SEA, seg_pos={0 0}, upload=55729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: context canceled; context canceled Jul 26 21:45:57.390037898 (58h10m40.234839421s) [seg_pos={0 0}, upload=55729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:45:57.390122120 (58h10m40.234923643s) [upload=55729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:45:57.390126761 (58h10m40.234928283s) [upload=55729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:45:57.431676820 (58h10m40.276478342s) [upload=55729]: Done uploading segments... Jul 26 21:45:57.431696074 (58h10m40.276497588s) [upload=55729]: Upload finished. err: <nil> Jul 26 21:46:59.287154075 (58h11m42.131955616s) [upload=55750]: Starting upload Jul 26 21:46:59.287242449 (58h11m42.132043975s) [upload=55750]: Done starting upload Jul 26 21:46:59.287323268 (58h11m42.132124793s) [upload=55750]: Uploading segments... Jul 26 21:46:59.287513138 (58h11m42.132314649s) [upload=55750]: Got next segment. Inline: false Jul 26 21:46:59.287551302 (58h11m42.132352824s) [upload=55750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:46:59.287556393 (58h11m42.132357916s) [upload=55750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:46:59.361463541 (58h11m42.206265069s) [seg_pos={0 0}, upload=55750]: Begin upload segment... Jul 26 21:46:59.361529577 (58h11m42.206331103s) [seg_pos={0 0}, upload=55750]: Done begin upload segment. Jul 26 21:46:59.361558285 (58h11m42.206359818s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.361600622 (58h11m42.206402155s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.362187253 (58h11m42.206988785s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363166949 (58h11m42.207968483s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363513513 (58h11m42.208315044s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364230475 (58h11m42.20903201s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364469220 (58h11m42.209270751s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365232523 (58h11m42.210034048s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365445039 (58h11m42.210246571s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366030012 (58h11m42.210831542s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366567327 (58h11m42.21136886s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367187629 (58h11m42.211989161s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367210599 (58h11m42.212012136s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367615629 (58h11m42.21241716s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367676352 (58h11m42.212477893s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.368021307 (58h11m42.21282284s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.650506128 (58h11m42.495307641s) [upload=55750]: Next returned nil segment Jul 26 21:46:59.650523082 (58h11m42.495324606s) [upload=55750]: Waiting for error group managing segments... Jul 26 21:46:59.696865628 (58h11m42.541667145s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.697705859 (58h11m42.542507374s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698162779 (58h11m42.542964296s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698588121 (58h11m42.543389635s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698986722 (58h11m42.54378823s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699489803 (58h11m42.544291314s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699968812 (58h11m42.544770338s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.707223719 (58h11m42.55202523s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.708984847 (58h11m42.553786361s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.710649854 (58h11m42.555451377s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.715423554 (58h11m42.56022508s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:12.547575970 (58h11m55.392377494s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:29.253110224 (58h12m12.097911758s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253197097 (58h12m12.097998686s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:38942->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253230948 (58h12m12.098032478s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253280622 (58h12m12.098082149s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253322073 (58h12m12.098123607s) [seg_pos={0 0}, upload=55750]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:47:29.276472271 (58h12m12.121273798s) [upload=55750]: Done uploading segments... Jul 26 21:47:29.276530550 (58h12m12.121332083s) [upload=55750]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.938726317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.042780815s FAIL (2025-07-26 21:51:03 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== _pos={0 0}, upload=55799]: Uploading piece... Jul 26 21:50:03.885866050 (58h14m46.730667582s) [noise=true , address=104.36.82.2:4739 , piece_id=KQIJXT2DGA674NNHT5I62JE4EJHJOF5XKWACD35AU7AXNQCJTYXA, seg_pos={0 0}, upload=55799]: Uploading piece... Jul 26 21:50:03.886290581 (58h14m46.731092126s) [noise=true , address=104.36.82.54:4177 , piece_id=XPMJWB7ETOU3LJFF6GHNFT7KYVTZSEVADVZZTW6J2FWM4DOKGUNA, seg_pos={0 0}, upload=55799]: Uploading piece... Jul 26 21:50:03.886692751 (58h14m46.731494286s) [noise=true , address=104.36.82.36:4647 , piece_id=FF7IIAA7QZWIFWFKGZEXBCQ4ZWDAC6ZGFYYIMDBNDUXFEANYIXZQ, seg_pos={0 0}, upload=55799]: Uploading piece... Jul 26 21:50:03.889061253 (58h14m46.733862788s) [noise=true , address=104.36.82.11:4139 , piece_id=BIC2AW74NOMJE4FYFORJE22GPOPW2KFDA6FH5A4IM75QW5APLMNA, seg_pos={0 0}, upload=55799]: Uploading piece... Jul 26 21:50:03.894470599 (58h14m46.739272136s) [noise=true , address=104.36.82.47:4552 , piece_id=ROYI7GKPYSCSJI5PNH3NNBZWMTWOZSHV6EMFQVWYT6PLFS7VNKUQ, seg_pos={0 0}, upload=55799]: Uploading piece... Jul 26 21:50:03.895673765 (58h14m46.740475301s) [noise=true , address=104.36.82.55:4209 , piece_id=7IWVEUAZVHOEMQNUG4F5Q7IH6ULJQTKBJJDWTMIJIHXB7VTANC5A, seg_pos={0 0}, upload=55799]: Uploading piece... Jul 26 21:50:03.897237619 (58h14m46.742039156s) [noise=true , address=104.36.82.53:4147 , piece_id=F7BSXV3SGP6VFKOXBCMCRF35ZVPEXDKKIKJ5ZN6VZJ3JT3JFN4AQ, seg_pos={0 0}, upload=55799]: Uploading piece... Jul 26 21:50:03.933705339 (58h14m46.778506874s) [noise=true , address=77.90.31.13:4644 , piece_id=AZN7OWRETT66S5A4BERCP5DGB6SRRATOHCW64CQCIICMLTUNKIYA, seg_pos={0 0}, upload=55799]: Uploading piece... Jul 26 21:50:03.934409815 (58h14m46.779211343s) [noise=true , address=104.36.82.16:4305 , piece_id=WFYNBB45Y7K3M4E6PWJNMUTNYZ6VDPSL4O4YBTVBH2WSVPAPEGVA, seg_pos={0 0}, upload=55799]: Uploading piece... Jul 26 21:50:03.950963237 (58h14m46.795764771s) [noise=true , address=104.36.82.9:4030 , piece_id=SQWZ6VQF7A4EK2CFYEWTQO7XRK4MR6OSSLSR3VY57IHGJ2CAP5ZA, seg_pos={0 0}, upload=55799]: Uploading piece... Jul 26 21:50:04.111803430 (58h14m46.956604965s) [upload=55799]: Next returned nil segment Jul 26 21:50:04.111827767 (58h14m46.956629305s) [upload=55799]: Waiting for error group managing segments... Jul 26 21:50:04.127465321 (58h14m46.97226683s) [noise=true , address=79.127.219.38:4005 , piece_id=HU72CVFNIKBTVPJ42PNNDFUWPIRLSGUZ3O6ISDYXNIAPEBUUNCFQ, seg_pos={0 0}, upload=55799]: Done uploading piece. err: <nil> Jul 26 21:50:05.417754398 (58h14m48.262555907s) [noise=true , address=77.90.31.13:4644 , piece_id=AZN7OWRETT66S5A4BERCP5DGB6SRRATOHCW64CQCIICMLTUNKIYA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: <nil> Jul 26 21:50:06.709651030 (58h14m49.554452542s) [noise=true , address=79.127.219.51:4201 , piece_id=UDTMYEHKB6V7BVKTENJDUSFR6PA3FQHV7D2PS4C2GQR6S2GZ4OFA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: <nil> Jul 26 21:51:03.729177306 (58h15m46.57397883s) [noise=true , address=104.36.82.15:4273 , piece_id=Q6JRGBVIA4NJFSUYY2SPSIKBQGMJ6I6UHXSMRFIZUBVBOYVY7VFA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:1cgTYjzrDTV5a4aohnpZvGPRweYkRv2dyKhQw46nQwMtAK3Kuv, address:104.36.82.15:4273): protocol: rpc: dial tcp 104.36.82.15:4273: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.729273695 (58h15m46.574075233s) [noise=true , address=104.36.82.2:4739 , piece_id=KQIJXT2DGA674NNHT5I62JE4EJHJOF5XKWACD35AU7AXNQCJTYXA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:1eEbhCXRGArodr2pffqXQxRR8w6u4c6J6auDbsCwe1NH5Wwycd, address:104.36.82.2:4739): protocol: rpc: dial tcp 104.36.82.2:4739: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.729424822 (58h15m46.574226351s) [noise=true , address=104.36.82.36:4647 , piece_id=FF7IIAA7QZWIFWFKGZEXBCQ4ZWDAC6ZGFYYIMDBNDUXFEANYIXZQ, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:1arYSsvaPDeLVdppvVEB5VYwRAcJkLfYWXMG7CAHvvY4YTKxpZ, address:104.36.82.36:4647): protocol: rpc: dial tcp 104.36.82.36:4647: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.729581352 (58h15m46.574382884s) [noise=true , address=104.36.82.9:4030 , piece_id=SQWZ6VQF7A4EK2CFYEWTQO7XRK4MR6OSSLSR3VY57IHGJ2CAP5ZA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:1MGCEjVE57ej5VuXgC6pYPFNUuAL851p2gqC53xxj6hEXvgYM5, address:104.36.82.9:4030): protocol: rpc: dial tcp 104.36.82.9:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.729647040 (58h15m46.574448571s) [noise=true , address=104.36.82.56:4230 , piece_id=2YRIE3FUO65QHQ2AVEFDC7O6OBBEJHLQN5O3WZEPBZE6ZCG4KTZA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: rpc: dial tcp 104.36.82.56:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.729713599 (58h15m46.574515126s) [noise=true , address=104.36.82.11:4139 , piece_id=BIC2AW74NOMJE4FYFORJE22GPOPW2KFDA6FH5A4IM75QW5APLMNA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:4139): protocol: rpc: dial tcp 104.36.82.11:4139: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.729770034 (58h15m46.574571562s) [noise=true , address=104.36.82.55:4209 , piece_id=7IWVEUAZVHOEMQNUG4F5Q7IH6ULJQTKBJJDWTMIJIHXB7VTANC5A, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR, address:104.36.82.55:4209): protocol: rpc: dial tcp 104.36.82.55:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.729836032 (58h15m46.574637558s) [noise=true , address=104.36.82.16:4305 , piece_id=WFYNBB45Y7K3M4E6PWJNMUTNYZ6VDPSL4O4YBTVBH2WSVPAPEGVA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:12ZMhNJYCNg8K16ZqsVTLBCwGn51vtG5u56cxcqgZPfQaJxgNZv, address:104.36.82.16:4305): protocol: rpc: dial tcp 104.36.82.16:4305: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.729891609 (58h15m46.574693135s) [noise=true , address=104.36.82.46:4499 , piece_id=55BXZHSQT36VSY6HQPUBOJC3PQJ7O32ILCPEEKGA5NSOG32FPIIA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: write tcp 159.223.143.39:58880->104.36.82.46:4499: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.730897660 (58h15m46.575699193s) [noise=true , address=104.36.82.7:4924 , piece_id=DFBH7M5QDLACEGIAWPDQAKWGFH643YDTL2CD36VJ6ZLLLLNQMCRA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:4924): protocol: rpc: dial tcp 104.36.82.7:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.731050606 (58h15m46.575852116s) [noise=true , address=104.36.82.54:4177 , piece_id=XPMJWB7ETOU3LJFF6GHNFT7KYVTZSEVADVZZTW6J2FWM4DOKGUNA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: rpc: dial tcp 104.36.82.54:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.731166804 (58h15m46.575968588s) [noise=true , address=104.36.82.47:4552 , piece_id=ROYI7GKPYSCSJI5PNH3NNBZWMTWOZSHV6EMFQVWYT6PLFS7VNKUQ, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.731273973 (58h15m46.576075487s) [noise=true , address=104.36.82.53:4147 , piece_id=F7BSXV3SGP6VFKOXBCMCRF35ZVPEXDKKIKJ5ZN6VZJ3JT3JFN4AQ, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: rpc: dial tcp 104.36.82.53:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.731416088 (58h15m46.576217645s) [noise=true , address=104.36.82.44:4890 , piece_id=3EXOBLE7L7ELF2JYULTFJUHBAF33Q2FZMVWDA4OJP54P5CHK54TQ, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.731450527 (58h15m46.57625205s) [noise=true , address=104.36.82.52:4093 , piece_id=I2EJM7YXXSYFYBMNMWWUQASP3TTYLMI7WE4T4GG33DGSFPV7HEJA, seg_pos={0 0}, upload=55799]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: rpc: dial tcp 104.36.82.52:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:51:03.731478417 (58h15m46.576279952s) [seg_pos={0 0}, upload=55799]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 26 21:51:03.766748558 (58h15m46.611550105s) [upload=55799]: Done uploading segments... Jul 26 21:51:03.766780514 (58h15m46.611582048s) [upload=55799]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.042867567s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................X.XX...X.X...X....] duration: 6.647635151s FAIL (2025-07-26 21:50:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 57.390018434 (58h10m40.234819956s) [noise=true , address=104.36.82.11:4115 , piece_id=OJKMPUDNWYKMQOUQEYRDZC2CLCSRDJ56K25VRIM6KKTMQGIY6SEA, seg_pos={0 0}, upload=55729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: context canceled; context canceled Jul 26 21:45:57.390037898 (58h10m40.234839421s) [seg_pos={0 0}, upload=55729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:45:57.390122120 (58h10m40.234923643s) [upload=55729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:45:57.390126761 (58h10m40.234928283s) [upload=55729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:45:57.431676820 (58h10m40.276478342s) [upload=55729]: Done uploading segments... Jul 26 21:45:57.431696074 (58h10m40.276497588s) [upload=55729]: Upload finished. err: <nil> Jul 26 21:46:59.287154075 (58h11m42.131955616s) [upload=55750]: Starting upload Jul 26 21:46:59.287242449 (58h11m42.132043975s) [upload=55750]: Done starting upload Jul 26 21:46:59.287323268 (58h11m42.132124793s) [upload=55750]: Uploading segments... Jul 26 21:46:59.287513138 (58h11m42.132314649s) [upload=55750]: Got next segment. Inline: false Jul 26 21:46:59.287551302 (58h11m42.132352824s) [upload=55750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:46:59.287556393 (58h11m42.132357916s) [upload=55750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:46:59.361463541 (58h11m42.206265069s) [seg_pos={0 0}, upload=55750]: Begin upload segment... Jul 26 21:46:59.361529577 (58h11m42.206331103s) [seg_pos={0 0}, upload=55750]: Done begin upload segment. Jul 26 21:46:59.361558285 (58h11m42.206359818s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.361600622 (58h11m42.206402155s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.362187253 (58h11m42.206988785s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363166949 (58h11m42.207968483s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363513513 (58h11m42.208315044s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364230475 (58h11m42.20903201s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364469220 (58h11m42.209270751s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365232523 (58h11m42.210034048s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365445039 (58h11m42.210246571s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366030012 (58h11m42.210831542s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366567327 (58h11m42.21136886s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367187629 (58h11m42.211989161s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367210599 (58h11m42.212012136s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367615629 (58h11m42.21241716s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367676352 (58h11m42.212477893s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.368021307 (58h11m42.21282284s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.650506128 (58h11m42.495307641s) [upload=55750]: Next returned nil segment Jul 26 21:46:59.650523082 (58h11m42.495324606s) [upload=55750]: Waiting for error group managing segments... Jul 26 21:46:59.696865628 (58h11m42.541667145s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.697705859 (58h11m42.542507374s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698162779 (58h11m42.542964296s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698588121 (58h11m42.543389635s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698986722 (58h11m42.54378823s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699489803 (58h11m42.544291314s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699968812 (58h11m42.544770338s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.707223719 (58h11m42.55202523s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.708984847 (58h11m42.553786361s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.710649854 (58h11m42.555451377s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.715423554 (58h11m42.56022508s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:12.547575970 (58h11m55.392377494s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:29.253110224 (58h12m12.097911758s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253197097 (58h12m12.097998686s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:38942->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253230948 (58h12m12.098032478s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253280622 (58h12m12.098082149s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253322073 (58h12m12.098123607s) [seg_pos={0 0}, upload=55750]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:47:29.276472271 (58h12m12.121273798s) [upload=55750]: Done uploading segments... Jul 26 21:47:29.276530550 (58h12m12.121332083s) [upload=55750]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.647677714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................X.......X.......X...X.....] duration: 6.746430677s FAIL (2025-07-26 21:50:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.746458439s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.053058196s FAIL (2025-07-26 21:50:03 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== e , address=104.36.82.54:4185 , piece_id=ZZ3VZWK6VB6HIU6ONP4SKNOV7N3LV45VCFP4WU4YE45PPNCOBHCQ, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.776874174 (58h13m46.621675706s) [noise=true , address=79.127.201.215:4358 , piece_id=6IWDDOCISW6G7KDI6TATGP3SMZ353KEV5RVJ2USFASQZPLTF4EBQ, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.777046331 (58h13m46.62184786s) [noise=true , address=77.90.31.13:4658 , piece_id=KMJJPTURGT46POQCPN7G3T5PBMLSLI4RGW3FDEHVVZ33NCIFEPKA, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.777337649 (58h13m46.622139176s) [noise=true , address=104.36.82.47:4543 , piece_id=JLGSJFB2FNI7N7PIMZQ4AVSU5RRMH5ZTV3LMO53II7DCSHOY5XYA, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.777490996 (58h13m46.622292532s) [noise=true , address=104.36.82.55:4195 , piece_id=ERJ3SJO5FXSMANKEPMU56F3HXKQNZTF6WXNAE23HYGP3FEZJXX7Q, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.777757374 (58h13m46.622558907s) [noise=true , address=104.36.82.14:4240 , piece_id=EC2M2NEP73JU63SYB2QDRFX64RAE4SHDFRDTKU463SH55T3BTLOQ, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.777901926 (58h13m46.622703455s) [noise=true , address=104.36.82.52:4087 , piece_id=5ZXRT2KC5RPQLYY5LSLHFKRZFEWQMUPDSUZTP4ZZHUH4D3ECJYOA, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.778365988 (58h13m46.62316752s) [noise=true , address=104.36.82.37:4687 , piece_id=ZCV6RONH4DG435M4JMDQJ4FWP46DLQLJGNFXUVBWTYOYA25IXV4Q, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.778740408 (58h13m46.623541944s) [noise=true , address=104.36.82.19:4402 , piece_id=IBJ3UM4BIA2STC6XBQ53FQERJUJIY2GCXEF6DP5WZCAB5TKIF76A, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.779200874 (58h13m46.624002412s) [noise=true , address=104.36.82.41:4785 , piece_id=PXHLCSKGYMBSFGJWOXBKLFJPH2INDPTBIXH3AK7H4DSVD25B7G4Q, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.779624500 (58h13m46.624426041s) [noise=true , address=79.127.205.245:4251 , piece_id=M6UEKWPOBC2H545MGNS453SKVZRGIJWSWJV4Z32L6SJXWXXPVY2Q, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.780034031 (58h13m46.624835569s) [noise=true , address=104.36.82.9:4019 , piece_id=3AGFBZ7OELY6J2BYS2BUYNSHSXNDSS7FVHGYRY2QJN3AJ6IE6LQA, seg_pos={0 0}, upload=55780]: Uploading piece... Jul 26 21:49:03.986887655 (58h13m46.831689168s) [upload=55780]: Next returned nil segment Jul 26 21:49:03.986905448 (58h13m46.831706977s) [upload=55780]: Waiting for error group managing segments... Jul 26 21:49:04.050473359 (58h13m46.895274886s) [noise=true , address=79.127.201.215:4358 , piece_id=6IWDDOCISW6G7KDI6TATGP3SMZ353KEV5RVJ2USFASQZPLTF4EBQ, seg_pos={0 0}, upload=55780]: Done uploading piece. err: <nil> Jul 26 21:49:04.420514746 (58h13m47.265316273s) [noise=true , address=79.127.205.234:4604 , piece_id=52ULFUA7ZU3Q5UVG6MKWZNN76PHMJ2NVNFTULWOKWQH4V7VYRV3A, seg_pos={0 0}, upload=55780]: Done uploading piece. err: <nil> Jul 26 21:49:04.524038675 (58h13m47.368840204s) [noise=true , address=79.127.205.245:4251 , piece_id=M6UEKWPOBC2H545MGNS453SKVZRGIJWSWJV4Z32L6SJXWXXPVY2Q, seg_pos={0 0}, upload=55780]: Done uploading piece. err: <nil> Jul 26 21:49:08.951691470 (58h13m51.796492985s) [noise=true , address=77.90.31.13:4658 , piece_id=KMJJPTURGT46POQCPN7G3T5PBMLSLI4RGW3FDEHVVZ33NCIFEPKA, seg_pos={0 0}, upload=55780]: Done uploading piece. err: <nil> Jul 26 21:50:03.606884100 (58h14m46.451685628s) [noise=true , address=104.36.82.46:4481 , piece_id=KT2Q4QFB46XXOOZOJRTAYHMSMM5IPTCAOY4JIKXZGDWOXGGB57OQ, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: write tcp 159.223.143.39:57996->104.36.82.46:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:50:03.607261296 (58h14m46.452062805s) [noise=true , address=104.36.82.54:4185 , piece_id=ZZ3VZWK6VB6HIU6ONP4SKNOV7N3LV45VCFP4WU4YE45PPNCOBHCQ, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:1cLdyyVLrp1yBmgsfBrfYc6N2RFbBichYoBbo2F7EZfaBXtkfk, address:104.36.82.54:4185): protocol: write tcp 159.223.143.39:36702->104.36.82.54:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:50:03.607358803 (58h14m46.452160331s) [noise=true , address=104.36.82.55:4195 , piece_id=ERJ3SJO5FXSMANKEPMU56F3HXKQNZTF6WXNAE23HYGP3FEZJXX7Q, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: write tcp 159.223.143.39:37044->104.36.82.55:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:50:03.607441267 (58h14m46.452242793s) [noise=true , address=104.36.82.52:4087 , piece_id=5ZXRT2KC5RPQLYY5LSLHFKRZFEWQMUPDSUZTP4ZZHUH4D3ECJYOA, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: write tcp 159.223.143.39:52918->104.36.82.52:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:50:03.607521556 (58h14m46.45232308s) [noise=true , address=104.36.82.37:4687 , piece_id=ZCV6RONH4DG435M4JMDQJ4FWP46DLQLJGNFXUVBWTYOYA25IXV4Q, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:12JZW19SfSwU95frntTgqDhiam1Rva1U8V7yfQkqtKYWntJ24Qf, address:104.36.82.37:4687): protocol: write tcp 159.223.143.39:37640->104.36.82.37:4687: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:50:03.621704593 (58h14m46.466506124s) [noise=true , address=104.36.82.53:4126 , piece_id=LZY4YEDS2KIXS2EFKMNEL5BLJEENIDC2UF6V2V6NGK7W4GZLBLSA, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:50:03.621973088 (58h14m46.466774616s) [noise=true , address=104.36.82.9:4019 , piece_id=3AGFBZ7OELY6J2BYS2BUYNSHSXNDSS7FVHGYRY2QJN3AJ6IE6LQA, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:12JuQKLQnXiEcyCj6xthEXSPy2Wu5T1B9m4q9YteoK69rMqVydD, address:104.36.82.9:4019): protocol: rpc: dial tcp 104.36.82.9:4019: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:50:03.622076856 (58h14m46.466878384s) [noise=true , address=104.36.82.14:4240 , piece_id=EC2M2NEP73JU63SYB2QDRFX64RAE4SHDFRDTKU463SH55T3BTLOQ, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:13SaC5StbguPT3abAT9oTX3mZAyGhJcpAA6b7ueyiDFHa8CCEo, address:104.36.82.14:4240): protocol: rpc: dial tcp 104.36.82.14:4240: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:50:03.622177409 (58h14m46.466978941s) [noise=true , address=104.36.82.40:4967 , piece_id=HG7NDLCBK4EX67GI2HZZ7GLCHDSFF7WEUDVIQRITKF6ZA5O6OQRA, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: write tcp 159.223.143.39:37860->104.36.82.40:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:50:03.622364639 (58h14m46.467166167s) [noise=true , address=104.36.82.19:4402 , piece_id=IBJ3UM4BIA2STC6XBQ53FQERJUJIY2GCXEF6DP5WZCAB5TKIF76A, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:104.36.82.19:4402): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:50:03.622584991 (58h14m46.467386518s) [noise=true , address=104.36.82.47:4543 , piece_id=JLGSJFB2FNI7N7PIMZQ4AVSU5RRMH5ZTV3LMO53II7DCSHOY5XYA, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:50:03.623558980 (58h14m46.468360506s) [noise=true , address=104.36.82.9:4032 , piece_id=3UUEXFFLJACXQI5HT2GLZ2EQCEI5A5BUXORCF4KROUIZZKWVLNIA, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ, address:104.36.82.9:4032): protocol: rpc: dial tcp 104.36.82.9:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:50:03.623703006 (58h14m46.468504537s) [noise=true , address=104.36.82.41:4785 , piece_id=PXHLCSKGYMBSFGJWOXBKLFJPH2INDPTBIXH3AK7H4DSVD25B7G4Q, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB, address:104.36.82.41:4785): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:50:03.623940751 (58h14m46.468742281s) [noise=true , address=104.36.82.56:4242 , piece_id=Z7ZS22K7IETRZLN7L6I2YASG6WJMAOQPICXVHX7AYLZUANG75LBA, seg_pos={0 0}, upload=55780]: Done uploading piece. err: ecclient: upload failed (node:1Q615w5XWJT1ownaJL1JFqyqntzMj6fgNieAfR6z421YCHuR5r, address:104.36.82.56:4242): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:50:03.624023210 (58h14m46.468824743s) [seg_pos={0 0}, upload=55780]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 21:50:03.653710089 (58h14m46.49851162s) [upload=55780]: Done uploading segments... Jul 26 21:50:03.653762514 (58h14m46.498564042s) [upload=55780]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.053087342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X.......X.......X...X....] duration: 9.054548414s FAIL (2025-07-26 21:49:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.054575681s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................................X.XX...X.X...X...] duration: 5.357842612s FAIL (2025-07-26 21:49:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 57.390018434 (58h10m40.234819956s) [noise=true , address=104.36.82.11:4115 , piece_id=OJKMPUDNWYKMQOUQEYRDZC2CLCSRDJ56K25VRIM6KKTMQGIY6SEA, seg_pos={0 0}, upload=55729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: context canceled; context canceled Jul 26 21:45:57.390037898 (58h10m40.234839421s) [seg_pos={0 0}, upload=55729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:45:57.390122120 (58h10m40.234923643s) [upload=55729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:45:57.390126761 (58h10m40.234928283s) [upload=55729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:45:57.431676820 (58h10m40.276478342s) [upload=55729]: Done uploading segments... Jul 26 21:45:57.431696074 (58h10m40.276497588s) [upload=55729]: Upload finished. err: <nil> Jul 26 21:46:59.287154075 (58h11m42.131955616s) [upload=55750]: Starting upload Jul 26 21:46:59.287242449 (58h11m42.132043975s) [upload=55750]: Done starting upload Jul 26 21:46:59.287323268 (58h11m42.132124793s) [upload=55750]: Uploading segments... Jul 26 21:46:59.287513138 (58h11m42.132314649s) [upload=55750]: Got next segment. Inline: false Jul 26 21:46:59.287551302 (58h11m42.132352824s) [upload=55750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:46:59.287556393 (58h11m42.132357916s) [upload=55750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:46:59.361463541 (58h11m42.206265069s) [seg_pos={0 0}, upload=55750]: Begin upload segment... Jul 26 21:46:59.361529577 (58h11m42.206331103s) [seg_pos={0 0}, upload=55750]: Done begin upload segment. Jul 26 21:46:59.361558285 (58h11m42.206359818s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.361600622 (58h11m42.206402155s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.362187253 (58h11m42.206988785s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363166949 (58h11m42.207968483s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363513513 (58h11m42.208315044s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364230475 (58h11m42.20903201s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364469220 (58h11m42.209270751s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365232523 (58h11m42.210034048s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365445039 (58h11m42.210246571s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366030012 (58h11m42.210831542s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366567327 (58h11m42.21136886s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367187629 (58h11m42.211989161s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367210599 (58h11m42.212012136s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367615629 (58h11m42.21241716s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367676352 (58h11m42.212477893s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.368021307 (58h11m42.21282284s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.650506128 (58h11m42.495307641s) [upload=55750]: Next returned nil segment Jul 26 21:46:59.650523082 (58h11m42.495324606s) [upload=55750]: Waiting for error group managing segments... Jul 26 21:46:59.696865628 (58h11m42.541667145s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.697705859 (58h11m42.542507374s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698162779 (58h11m42.542964296s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698588121 (58h11m42.543389635s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698986722 (58h11m42.54378823s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699489803 (58h11m42.544291314s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699968812 (58h11m42.544770338s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.707223719 (58h11m42.55202523s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.708984847 (58h11m42.553786361s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.710649854 (58h11m42.555451377s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.715423554 (58h11m42.56022508s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:12.547575970 (58h11m55.392377494s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:29.253110224 (58h12m12.097911758s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253197097 (58h12m12.097998686s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:38942->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253230948 (58h12m12.098032478s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253280622 (58h12m12.098082149s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253322073 (58h12m12.098123607s) [seg_pos={0 0}, upload=55750]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:47:29.276472271 (58h12m12.121273798s) [upload=55750]: Done uploading segments... Jul 26 21:47:29.276530550 (58h12m12.121332083s) [upload=55750]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.357866927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 1m0.03232751s FAIL (2025-07-26 21:48:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== h12m42.317431994s) [noise=true , address=104.36.82.54:4157 , piece_id=PKQOCCIFAR67SG2ZIRPQ52F2LEOXYCBRJVQ447LR7KXZSDWECV3A, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.472797875 (58h12m42.317599408s) [noise=true , address=79.127.201.215:4354 , piece_id=FINFZ2K5QAHY6IGR7Y23L5IMU77FM4FXTIWHSLKVZCRASVOAPKYQ, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.473018231 (58h12m42.317819764s) [noise=true , address=64.49.13.154:4928 , piece_id=CX7IMKHZTURLU5NG755KE6Z7ZBD26L54EN7MDWKZULYTVYB3WDFA, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.473313674 (58h12m42.318115209s) [noise=true , address=104.36.82.5:4860 , piece_id=BRWRMZ276LOELMEH6EDHRO2NYGAOCLQEZ6BZA4CNF4RHDIR7RMWA, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.473400113 (58h12m42.318201651s) [noise=true , address=104.36.82.2:4735 , piece_id=CMMYCT5IEDZMHWMTY6UIHZ77ZQEQ3AXRMDWYIMFEBIT3HKNKAWZA, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.473775560 (58h12m42.318577076s) [noise=true , address=104.36.82.40:4946 , piece_id=YJHQOKOXEMY2CDC2LMOX4MZAMGABA5MJZYXMQ47OYBUP3KQ7UC6Q, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.473823056 (58h12m42.31862459s) [noise=true , address=79.127.219.38:4008 , piece_id=45572ZRC2QJAPTKZEC6FGXTOF4V6CYRNHKGR6ED4BZ7X32FFCHZA, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.474329620 (58h12m42.319131156s) [noise=true , address=156.146.43.225:4448 , piece_id=QJYIQIO4O2643UDIXXRTGTU3PAOVO4G36G7XXX5PHOVKLG4WTAJA, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.474765459 (58h12m42.319566993s) [noise=true , address=104.36.82.52:4092 , piece_id=CPF7XQOALREOWGPVPEPUEJWCDPQJUGLIAGSHCJQ52HVAR3AKOZNQ, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.475281287 (58h12m42.320082823s) [noise=true , address=104.36.82.47:4544 , piece_id=MGSLI5ZOADGCNW76OHCQXUWY6DYDUTIA4CV36TTV3LQLWPYAFI4A, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.475710926 (58h12m42.32051246s) [noise=true , address=104.36.82.53:4151 , piece_id=24LBGWEBDMMMTE73ZSVWT5RCORKW67EXYI3WEA6QWAQUPRXGFRMA, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.476127274 (58h12m42.320928808s) [noise=true , address=104.36.82.14:4249 , piece_id=GWVXPWEUY2WUQQSBIXIN4MKJZD5NWFSXKKGBEPQNWVPWWIBCPVPA, seg_pos={0 0}, upload=55766]: Uploading piece... Jul 26 21:47:59.575374243 (58h12m42.420175771s) [upload=55766]: Next returned nil segment Jul 26 21:47:59.575395111 (58h12m42.420196643s) [upload=55766]: Waiting for error group managing segments... Jul 26 21:47:59.748482122 (58h12m42.593283636s) [noise=true , address=156.146.43.225:4448 , piece_id=QJYIQIO4O2643UDIXXRTGTU3PAOVO4G36G7XXX5PHOVKLG4WTAJA, seg_pos={0 0}, upload=55766]: Done uploading piece. err: <nil> Jul 26 21:47:59.770246584 (58h12m42.615048168s) [noise=true , address=79.127.201.214:4317 , piece_id=2Q7EVISY2T5YR5NPEXM22BU5ILDGZM4ZHVJKNAKHMNPFYO6YBOMA, seg_pos={0 0}, upload=55766]: Done uploading piece. err: <nil> Jul 26 21:47:59.835453475 (58h12m42.68025499s) [noise=true , address=79.127.219.38:4008 , piece_id=45572ZRC2QJAPTKZEC6FGXTOF4V6CYRNHKGR6ED4BZ7X32FFCHZA, seg_pos={0 0}, upload=55766]: Done uploading piece. err: <nil> Jul 26 21:47:59.890466187 (58h12m42.735267716s) [noise=true , address=156.146.43.225:4434 , piece_id=3TFLCZJLT6CBSSQCAFCL2CI6KGQFJ2RKTVIYGQJHG2VDLJVKPBBQ, seg_pos={0 0}, upload=55766]: Done uploading piece. err: <nil> Jul 26 21:48:00.154785761 (58h12m42.999587274s) [noise=true , address=64.49.13.154:4928 , piece_id=CX7IMKHZTURLU5NG755KE6Z7ZBD26L54EN7MDWKZULYTVYB3WDFA, seg_pos={0 0}, upload=55766]: Done uploading piece. err: <nil> Jul 26 21:48:06.055517901 (58h12m48.900319433s) [noise=true , address=79.127.201.215:4354 , piece_id=FINFZ2K5QAHY6IGR7Y23L5IMU77FM4FXTIWHSLKVZCRASVOAPKYQ, seg_pos={0 0}, upload=55766]: Done uploading piece. err: <nil> Jul 26 21:48:59.354403872 (58h13m42.199205394s) [noise=true , address=104.36.82.5:4872 , piece_id=HQRDPXCW4PG2JR2HMNO77HJICMOLZ6L5EAUXYDVEDCQWEX6CH2KQ, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:1zqxiFpqKeXT9mhXUDuZZmvJkhejC8gcHTQgTadY1xUpQyLyZi, address:104.36.82.5:4872): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:48:59.354894119 (58h13m42.199695642s) [noise=true , address=104.36.82.52:4092 , piece_id=CPF7XQOALREOWGPVPEPUEJWCDPQJUGLIAGSHCJQ52HVAR3AKOZNQ, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: write tcp 159.223.143.39:34994->104.36.82.52:4092: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:48:59.354972610 (58h13m42.199774133s) [noise=true , address=104.36.82.46:4496 , piece_id=NAIESVKAPOTPUZN4MFC5BO6HR3V7VE7H3SIF52SMSQQR43UXMLGQ, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: write tcp 159.223.143.39:47858->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:48:59.355065780 (58h13m42.199867303s) [noise=true , address=104.36.82.2:4735 , piece_id=CMMYCT5IEDZMHWMTY6UIHZ77ZQEQ3AXRMDWYIMFEBIT3HKNKAWZA, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:125ShGXckiJ3N5UWVmsMdhj98oFaRN7sg1KRucyUzqENW4Jp4kS, address:104.36.82.2:4735): protocol: write tcp 159.223.143.39:38842->104.36.82.2:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:48:59.355145671 (58h13m42.199947198s) [noise=true , address=104.36.82.47:4544 , piece_id=MGSLI5ZOADGCNW76OHCQXUWY6DYDUTIA4CV36TTV3LQLWPYAFI4A, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:104.36.82.47:4544): protocol: write tcp 159.223.143.39:33200->104.36.82.47:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:48:59.355411087 (58h13m42.200212608s) [noise=true , address=104.36.82.56:4237 , piece_id=5PVSTTRZAFIPH7UPMOQTKU2BLB7E74D3JEYMPEL7FYKT2EOXJQKQ, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:1VzewCRbNpJv7kNwp9wruQuoVSaZFjA44qN33Ct7qm1GnfhtDj, address:104.36.82.56:4237): protocol: write tcp 159.223.143.39:51190->104.36.82.56:4237: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:48:59.355496652 (58h13m42.200298175s) [noise=true , address=104.36.82.40:4946 , piece_id=YJHQOKOXEMY2CDC2LMOX4MZAMGABA5MJZYXMQ47OYBUP3KQ7UC6Q, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX, address:104.36.82.40:4946): protocol: write tcp 159.223.143.39:40464->104.36.82.40:4946: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:48:59.355569041 (58h13m42.200370629s) [noise=true , address=104.36.82.54:4157 , piece_id=PKQOCCIFAR67SG2ZIRPQ52F2LEOXYCBRJVQ447LR7KXZSDWECV3A, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: write tcp 159.223.143.39:60610->104.36.82.54:4157: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:48:59.355652011 (58h13m42.200453523s) [noise=true , address=104.36.82.53:4151 , piece_id=24LBGWEBDMMMTE73ZSVWT5RCORKW67EXYI3WEA6QWAQUPRXGFRMA, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:196fMe3fbxKaPgNs1zQx7Kg3Rh68s6Gtp7mbu3hn654WCbZctx, address:104.36.82.53:4151): protocol: write tcp 159.223.143.39:51516->104.36.82.53:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:48:59.355702514 (58h13m42.200504037s) [noise=true , address=104.36.82.5:4860 , piece_id=BRWRMZ276LOELMEH6EDHRO2NYGAOCLQEZ6BZA4CNF4RHDIR7RMWA, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:1Qr9fghoRVbAvNm9AMEHhhtc7JqHWBuZerBttg3KW3J3GcDjrE, address:104.36.82.5:4860): protocol: write tcp 159.223.143.39:35646->104.36.82.5:4860: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:48:59.355718408 (58h13m42.200519924s) [noise=true , address=104.36.82.14:4249 , piece_id=GWVXPWEUY2WUQQSBIXIN4MKJZD5NWFSXKKGBEPQNWVPWWIBCPVPA, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:1GnfZTc54L9eRyjCUwQuaFcG6FjsUDjDwT3Gx4zoNTQrvP158L, address:104.36.82.14:4249): protocol: write tcp 159.223.143.39:49346->104.36.82.14:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:48:59.355728034 (58h13m42.200529547s) [noise=true , address=104.36.82.11:4142 , piece_id=7PE6J24WHM57TZJTDIASUEQQ262F75CCC7AXITTCPRKM47VJ7SDA, seg_pos={0 0}, upload=55766]: Done uploading piece. err: ecclient: upload failed (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7, address:104.36.82.11:4142): protocol: write tcp 159.223.143.39:41952->104.36.82.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:48:59.355764973 (58h13m42.200566498s) [seg_pos={0 0}, upload=55766]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 21:48:59.383576140 (58h13m42.228377655s) [upload=55766]: Done uploading segments... Jul 26 21:48:59.383631289 (58h13m42.22843282s) [upload=55766]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.03236527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X.......X.......X...X...] duration: 9.714817615s FAIL (2025-07-26 21:48:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.714843697s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................................X.XX...X.X...X..] duration: 11.373994917s FAIL (2025-07-26 21:48:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 57.390018434 (58h10m40.234819956s) [noise=true , address=104.36.82.11:4115 , piece_id=OJKMPUDNWYKMQOUQEYRDZC2CLCSRDJ56K25VRIM6KKTMQGIY6SEA, seg_pos={0 0}, upload=55729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: context canceled; context canceled Jul 26 21:45:57.390037898 (58h10m40.234839421s) [seg_pos={0 0}, upload=55729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:45:57.390122120 (58h10m40.234923643s) [upload=55729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:45:57.390126761 (58h10m40.234928283s) [upload=55729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:45:57.431676820 (58h10m40.276478342s) [upload=55729]: Done uploading segments... Jul 26 21:45:57.431696074 (58h10m40.276497588s) [upload=55729]: Upload finished. err: <nil> Jul 26 21:46:59.287154075 (58h11m42.131955616s) [upload=55750]: Starting upload Jul 26 21:46:59.287242449 (58h11m42.132043975s) [upload=55750]: Done starting upload Jul 26 21:46:59.287323268 (58h11m42.132124793s) [upload=55750]: Uploading segments... Jul 26 21:46:59.287513138 (58h11m42.132314649s) [upload=55750]: Got next segment. Inline: false Jul 26 21:46:59.287551302 (58h11m42.132352824s) [upload=55750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:46:59.287556393 (58h11m42.132357916s) [upload=55750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:46:59.361463541 (58h11m42.206265069s) [seg_pos={0 0}, upload=55750]: Begin upload segment... Jul 26 21:46:59.361529577 (58h11m42.206331103s) [seg_pos={0 0}, upload=55750]: Done begin upload segment. Jul 26 21:46:59.361558285 (58h11m42.206359818s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.361600622 (58h11m42.206402155s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.362187253 (58h11m42.206988785s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363166949 (58h11m42.207968483s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363513513 (58h11m42.208315044s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364230475 (58h11m42.20903201s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364469220 (58h11m42.209270751s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365232523 (58h11m42.210034048s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365445039 (58h11m42.210246571s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366030012 (58h11m42.210831542s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366567327 (58h11m42.21136886s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367187629 (58h11m42.211989161s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367210599 (58h11m42.212012136s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367615629 (58h11m42.21241716s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367676352 (58h11m42.212477893s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.368021307 (58h11m42.21282284s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.650506128 (58h11m42.495307641s) [upload=55750]: Next returned nil segment Jul 26 21:46:59.650523082 (58h11m42.495324606s) [upload=55750]: Waiting for error group managing segments... Jul 26 21:46:59.696865628 (58h11m42.541667145s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.697705859 (58h11m42.542507374s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698162779 (58h11m42.542964296s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698588121 (58h11m42.543389635s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698986722 (58h11m42.54378823s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699489803 (58h11m42.544291314s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699968812 (58h11m42.544770338s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.707223719 (58h11m42.55202523s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.708984847 (58h11m42.553786361s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.710649854 (58h11m42.555451377s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.715423554 (58h11m42.56022508s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:12.547575970 (58h11m55.392377494s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:29.253110224 (58h12m12.097911758s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253197097 (58h12m12.097998686s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:38942->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253230948 (58h12m12.098032478s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253280622 (58h12m12.098082149s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253322073 (58h12m12.098123607s) [seg_pos={0 0}, upload=55750]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:47:29.276472271 (58h12m12.121273798s) [upload=55750]: Done uploading segments... Jul 26 21:47:29.276530550 (58h12m12.121332083s) [upload=55750]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.374032144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 1m0.041970419s FAIL (2025-07-26 21:47:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== ece_id=ND2ZZMRFHRFL576PVRIMIZNUVHFJV7MX3HZCQ3NNUGC5CMI6VVPQ, seg_pos={0 0}, upload=55749]: Uploading piece... Jul 26 21:46:59.376144288 (58h11m42.220945824s) [noise=true , address=104.36.82.15:4273 , piece_id=AZA7SK5AIA2OQEWJP5SDKLF2JZUWW5FOINGKFWCSVVL3GASHGBCA, seg_pos={0 0}, upload=55749]: Uploading piece... Jul 26 21:46:59.376599078 (58h11m42.221400607s) [noise=true , address=104.36.82.22:4516 , piece_id=DJFSCVZ5ZUIELV24HMOK4NXBFYAU5TVCUHGMBLXPTKZNSDQBRHMA, seg_pos={0 0}, upload=55749]: Uploading piece... Jul 26 21:46:59.376899822 (58h11m42.221701347s) [noise=true , address=104.36.82.40:4966 , piece_id=UREUEUXDVWOX75FUTKUFUWDDMKBXFNFCCK6RXXYKA6LVQQI6X7UQ, seg_pos={0 0}, upload=55749]: Uploading piece... Jul 26 21:46:59.377266085 (58h11m42.222067614s) [noise=true , address=104.36.82.14:4237 , piece_id=FJ5AWL6E7PXIPAGV2L3IP5UJ5QBUJA4Y72JGIKMAVQLUNLWO5GMQ, seg_pos={0 0}, upload=55749]: Uploading piece... Jul 26 21:46:59.377550288 (58h11m42.222351819s) [noise=true , address=104.36.82.55:4225 , piece_id=VC22R3GG7SJ4UEZYIMR6WSU26X75CSJJTSEUQ5TCJUKNKUHXO6PA, seg_pos={0 0}, upload=55749]: Uploading piece... Jul 26 21:46:59.378015221 (58h11m42.222816751s) [noise=true , address=77.90.31.18:4833 , piece_id=KPXQJIRV77PCTH2AAFASOHFE36UNXEEEYXNSGHHV66TFVCQTZEOQ, seg_pos={0 0}, upload=55749]: Uploading piece... Jul 26 21:46:59.378402794 (58h11m42.223204319s) [noise=true , address=104.36.82.50:4048 , piece_id=BQQD47NWQHFZICFWLMZNWF2EJKFEL3OSQMPSEHDW54UHWXC4ZDLA, seg_pos={0 0}, upload=55749]: Uploading piece... Jul 26 21:46:59.378832066 (58h11m42.223633598s) [noise=true , address=104.36.82.54:4192 , piece_id=DIAXGIWY2ZBUB453XZO7VG5HA27RQGRVXJA4W74OMH7VL3YQTWVQ, seg_pos={0 0}, upload=55749]: Uploading piece... Jul 26 21:46:59.379238827 (58h11m42.224040355s) [noise=true , address=104.36.82.46:4482 , piece_id=IJGY5YTVWC4KV2XSVCYXVZADSDD6PMEMMVWYQFXXACPMFUKULK6A, seg_pos={0 0}, upload=55749]: Uploading piece... Jul 26 21:46:59.379598488 (58h11m42.224400016s) [noise=true , address=156.146.43.225:4448 , piece_id=3A273RID4PYZHQUZFT3H6TPACY6FGNFNJWELJPWO26GTUKOMFX2A, seg_pos={0 0}, upload=55749]: Uploading piece... Jul 26 21:46:59.778443670 (58h11m42.623245203s) [upload=55749]: Next returned nil segment Jul 26 21:46:59.778465829 (58h11m42.623267364s) [upload=55749]: Waiting for error group managing segments... Jul 26 21:46:59.956994019 (58h11m42.801795535s) [noise=true , address=156.146.43.225:4448 , piece_id=3A273RID4PYZHQUZFT3H6TPACY6FGNFNJWELJPWO26GTUKOMFX2A, seg_pos={0 0}, upload=55749]: Done uploading piece. err: <nil> Jul 26 21:47:02.048963483 (58h11m44.893765006s) [noise=true , address=77.90.31.18:4833 , piece_id=KPXQJIRV77PCTH2AAFASOHFE36UNXEEEYXNSGHHV66TFVCQTZEOQ, seg_pos={0 0}, upload=55749]: Done uploading piece. err: <nil> Jul 26 21:47:03.079100780 (58h11m45.923902311s) [noise=true , address=79.127.201.215:4355 , piece_id=VJRSAECJG23YH3WISJL3BTLBZTUFAG6HHTXQBXE4PWE7NTGIUBUA, seg_pos={0 0}, upload=55749]: Done uploading piece. err: <nil> Jul 26 21:47:59.252718624 (58h12m42.097520153s) [noise=true , address=104.36.82.53:4146 , piece_id=5OB7KNZ332ZVT55UTVZQOQ46DHRR6LAOJDMYCHL4FGTQ45AIG77A, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: rpc: dial tcp 104.36.82.53:4146: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.252874896 (58h12m42.097676404s) [noise=true , address=104.36.82.55:4225 , piece_id=VC22R3GG7SJ4UEZYIMR6WSU26X75CSJJTSEUQ5TCJUKNKUHXO6PA, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: rpc: dial tcp 104.36.82.55:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.253681622 (58h12m42.098483134s) [noise=true , address=104.36.82.54:4192 , piece_id=DIAXGIWY2ZBUB453XZO7VG5HA27RQGRVXJA4W74OMH7VL3YQTWVQ, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY, address:104.36.82.54:4192): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:47:59.254526434 (58h12m42.099327946s) [noise=true , address=104.36.82.47:4522 , piece_id=YLI425KQZIUPMYPJ5RIIXIIHYM4QRFQAPRKJUK64NT33O6KRYXPA, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: rpc: dial tcp 104.36.82.47:4522: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.254578976 (58h12m42.099380489s) [noise=true , address=104.36.82.12:4162 , piece_id=EWG4BJH6YMN73YEN7Y4F7JLSELDKJJ3ZSHUCNSFQWCHNRRS2PBIQ, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:1yxXQkfWVGwBuptpPjFgjhfTraAmE3WqtS3WEDbhQsJftjeaxV, address:104.36.82.12:4162): protocol: rpc: dial tcp 104.36.82.12:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.254634741 (58h12m42.099436253s) [noise=true , address=104.36.82.15:4265 , piece_id=ND2ZZMRFHRFL576PVRIMIZNUVHFJV7MX3HZCQ3NNUGC5CMI6VVPQ, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:1NscbwYiKarBdony9TwdccuEKg7f4VXEXerFfWmCiat4RKdXFi, address:104.36.82.15:4265): protocol: rpc: dial tcp 104.36.82.15:4265: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.254709954 (58h12m42.099511467s) [noise=true , address=104.36.82.40:4966 , piece_id=UREUEUXDVWOX75FUTKUFUWDDMKBXFNFCCK6RXXYKA6LVQQI6X7UQ, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:104.36.82.40:4966): protocol: rpc: dial tcp 104.36.82.40:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.254759048 (58h12m42.099560559s) [noise=true , address=104.36.82.14:4237 , piece_id=FJ5AWL6E7PXIPAGV2L3IP5UJ5QBUJA4Y72JGIKMAVQLUNLWO5GMQ, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:104.36.82.14:4237): protocol: rpc: dial tcp 104.36.82.14:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.254804270 (58h12m42.099605783s) [noise=true , address=104.36.82.39:4747 , piece_id=LIRPYOW4QC4R3JDZRBAZXJFO5YCDA3W3KRD27RTZFOKCV7GZ74WA, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:1bmbaxrj3X67qtdL5boUEPGBsujt1ggcGcGsJvvGTAMWoQe8Qg, address:104.36.82.39:4747): protocol: write tcp 159.223.143.39:50244->104.36.82.39:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.255535310 (58h12m42.100336823s) [noise=true , address=104.36.82.52:4111 , piece_id=ABWTSLXSQ2GCTUFTDFJ27GJMBHVXWIEZ6AVWTW62HQJVTPO5NMSQ, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: rpc: dial tcp 104.36.82.52:4111: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.255586982 (58h12m42.100388493s) [noise=true , address=104.36.82.22:4516 , piece_id=DJFSCVZ5ZUIELV24HMOK4NXBFYAU5TVCUHGMBLXPTKZNSDQBRHMA, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:12Lhy3AxzYoF25AxWdvjrrV5NNCwP1cDqkEvv4tz9t9KZmF8WPK, address:104.36.82.22:4516): protocol: rpc: dial tcp 104.36.82.22:4516: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.255630013 (58h12m42.100431528s) [noise=true , address=104.36.82.50:4048 , piece_id=BQQD47NWQHFZICFWLMZNWF2EJKFEL3OSQMPSEHDW54UHWXC4ZDLA, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:1rJd5FahWjKfvJstcoLE5uJGgtVj2uZirPpCJxbfGPGQ3j4qxC, address:104.36.82.50:4048): protocol: rpc: dial tcp 104.36.82.50:4048: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.255812246 (58h12m42.100613758s) [noise=true , address=104.36.82.46:4482 , piece_id=IJGY5YTVWC4KV2XSVCYXVZADSDD6PMEMMVWYQFXXACPMFUKULK6A, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:47:59.256351256 (58h12m42.101152777s) [noise=true , address=104.36.82.18:4385 , piece_id=PDTHG5DI7F3XRLALXSTRCHWNDCWZSLTUSNV7RCBGAMXIFNZUERJQ, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:123PZh1ieWYoc9U5CRrRzV1ZTpGYmjLmh5CpRy1NRC8wiLkumbs, address:104.36.82.18:4385): protocol: rpc: dial tcp 104.36.82.18:4385: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.256409338 (58h12m42.101210861s) [noise=true , address=104.36.82.15:4273 , piece_id=AZA7SK5AIA2OQEWJP5SDKLF2JZUWW5FOINGKFWCSVVL3GASHGBCA, seg_pos={0 0}, upload=55749]: Done uploading piece. err: ecclient: upload failed (node:1cgTYjzrDTV5a4aohnpZvGPRweYkRv2dyKhQw46nQwMtAK3Kuv, address:104.36.82.15:4273): protocol: write tcp 159.223.143.39:60922->104.36.82.15:4273: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:59.256434030 (58h12m42.101235552s) [seg_pos={0 0}, upload=55749]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 26 21:47:59.285651457 (58h12m42.130452972s) [upload=55749]: Done uploading segments... Jul 26 21:47:59.285681210 (58h12m42.130482732s) [upload=55749]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.041988481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................X.XX...X.X...X.] duration: 2.920826572s FAIL (2025-07-26 21:47:52 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 57.390018434 (58h10m40.234819956s) [noise=true , address=104.36.82.11:4115 , piece_id=OJKMPUDNWYKMQOUQEYRDZC2CLCSRDJ56K25VRIM6KKTMQGIY6SEA, seg_pos={0 0}, upload=55729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: context canceled; context canceled Jul 26 21:45:57.390037898 (58h10m40.234839421s) [seg_pos={0 0}, upload=55729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:45:57.390122120 (58h10m40.234923643s) [upload=55729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:45:57.390126761 (58h10m40.234928283s) [upload=55729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:45:57.431676820 (58h10m40.276478342s) [upload=55729]: Done uploading segments... Jul 26 21:45:57.431696074 (58h10m40.276497588s) [upload=55729]: Upload finished. err: <nil> Jul 26 21:46:59.287154075 (58h11m42.131955616s) [upload=55750]: Starting upload Jul 26 21:46:59.287242449 (58h11m42.132043975s) [upload=55750]: Done starting upload Jul 26 21:46:59.287323268 (58h11m42.132124793s) [upload=55750]: Uploading segments... Jul 26 21:46:59.287513138 (58h11m42.132314649s) [upload=55750]: Got next segment. Inline: false Jul 26 21:46:59.287551302 (58h11m42.132352824s) [upload=55750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:46:59.287556393 (58h11m42.132357916s) [upload=55750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:46:59.361463541 (58h11m42.206265069s) [seg_pos={0 0}, upload=55750]: Begin upload segment... Jul 26 21:46:59.361529577 (58h11m42.206331103s) [seg_pos={0 0}, upload=55750]: Done begin upload segment. Jul 26 21:46:59.361558285 (58h11m42.206359818s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.361600622 (58h11m42.206402155s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.362187253 (58h11m42.206988785s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363166949 (58h11m42.207968483s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363513513 (58h11m42.208315044s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364230475 (58h11m42.20903201s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364469220 (58h11m42.209270751s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365232523 (58h11m42.210034048s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365445039 (58h11m42.210246571s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366030012 (58h11m42.210831542s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366567327 (58h11m42.21136886s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367187629 (58h11m42.211989161s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367210599 (58h11m42.212012136s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367615629 (58h11m42.21241716s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367676352 (58h11m42.212477893s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.368021307 (58h11m42.21282284s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.650506128 (58h11m42.495307641s) [upload=55750]: Next returned nil segment Jul 26 21:46:59.650523082 (58h11m42.495324606s) [upload=55750]: Waiting for error group managing segments... Jul 26 21:46:59.696865628 (58h11m42.541667145s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.697705859 (58h11m42.542507374s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698162779 (58h11m42.542964296s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698588121 (58h11m42.543389635s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698986722 (58h11m42.54378823s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699489803 (58h11m42.544291314s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699968812 (58h11m42.544770338s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.707223719 (58h11m42.55202523s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.708984847 (58h11m42.553786361s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.710649854 (58h11m42.555451377s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.715423554 (58h11m42.56022508s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:12.547575970 (58h11m55.392377494s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:29.253110224 (58h12m12.097911758s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253197097 (58h12m12.097998686s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:38942->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253230948 (58h12m12.098032478s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253280622 (58h12m12.098082149s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253322073 (58h12m12.098123607s) [seg_pos={0 0}, upload=55750]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:47:29.276472271 (58h12m12.121273798s) [upload=55750]: Done uploading segments... Jul 26 21:47:29.276530550 (58h12m12.121332083s) [upload=55750]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.920857229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X.......X.......X...X..] duration: 17.695078749s FAIL (2025-07-26 21:47:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 17.695105508s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................X.XX...X.X...X] duration: 30.045114748s FAIL (2025-07-26 21:47:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 57.390018434 (58h10m40.234819956s) [noise=true , address=104.36.82.11:4115 , piece_id=OJKMPUDNWYKMQOUQEYRDZC2CLCSRDJ56K25VRIM6KKTMQGIY6SEA, seg_pos={0 0}, upload=55729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: context canceled; context canceled Jul 26 21:45:57.390037898 (58h10m40.234839421s) [seg_pos={0 0}, upload=55729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:45:57.390122120 (58h10m40.234923643s) [upload=55729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:45:57.390126761 (58h10m40.234928283s) [upload=55729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:45:57.431676820 (58h10m40.276478342s) [upload=55729]: Done uploading segments... Jul 26 21:45:57.431696074 (58h10m40.276497588s) [upload=55729]: Upload finished. err: <nil> Jul 26 21:46:59.287154075 (58h11m42.131955616s) [upload=55750]: Starting upload Jul 26 21:46:59.287242449 (58h11m42.132043975s) [upload=55750]: Done starting upload Jul 26 21:46:59.287323268 (58h11m42.132124793s) [upload=55750]: Uploading segments... Jul 26 21:46:59.287513138 (58h11m42.132314649s) [upload=55750]: Got next segment. Inline: false Jul 26 21:46:59.287551302 (58h11m42.132352824s) [upload=55750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:46:59.287556393 (58h11m42.132357916s) [upload=55750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:46:59.361463541 (58h11m42.206265069s) [seg_pos={0 0}, upload=55750]: Begin upload segment... Jul 26 21:46:59.361529577 (58h11m42.206331103s) [seg_pos={0 0}, upload=55750]: Done begin upload segment. Jul 26 21:46:59.361558285 (58h11m42.206359818s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.361600622 (58h11m42.206402155s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.362187253 (58h11m42.206988785s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363166949 (58h11m42.207968483s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.363513513 (58h11m42.208315044s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364230475 (58h11m42.20903201s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.364469220 (58h11m42.209270751s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365232523 (58h11m42.210034048s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.365445039 (58h11m42.210246571s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366030012 (58h11m42.210831542s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.366567327 (58h11m42.21136886s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367187629 (58h11m42.211989161s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367210599 (58h11m42.212012136s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367615629 (58h11m42.21241716s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.367676352 (58h11m42.212477893s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.368021307 (58h11m42.21282284s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Uploading piece... Jul 26 21:46:59.650506128 (58h11m42.495307641s) [upload=55750]: Next returned nil segment Jul 26 21:46:59.650523082 (58h11m42.495324606s) [upload=55750]: Waiting for error group managing segments... Jul 26 21:46:59.696865628 (58h11m42.541667145s) [noise=true , address=45.91.116.10:4083 , piece_id=OHKEPWC6WBUZY63VT44HTZ6ZICAL37AVMZQIXNLSRLWRCN5T7I6Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.697705859 (58h11m42.542507374s) [noise=true , address=45.91.116.7:4011 , piece_id=QOE6ZVFFJR2LSPZ25HU2KVBQRORDOZ4XMYSB3ZET7BVBNENOVGDQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698162779 (58h11m42.542964296s) [noise=true , address=45.91.116.7:4010 , piece_id=63GIGPHSKOHLEA3NXVEI2LOPVGIFX2WGEGDDTIPDBP37UYCCMR4Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698588121 (58h11m42.543389635s) [noise=true , address=45.91.116.12:4226 , piece_id=3QDKNMEFWNY6FS5QUG2HFIFDTNQTJKTSROSH5YQM4KT4HICCY36Q, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.698986722 (58h11m42.54378823s) [noise=true , address=45.91.116.14:4189 , piece_id=7QODTPP23NSBZXMNB6UYOGHLCUPWQPAAWWB3VPFB3GIJT2NSHKSA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699489803 (58h11m42.544291314s) [noise=true , address=45.91.116.12:4227 , piece_id=CFNMQTSYPLE5EYICJB4ENVHRTKIM7RSOSGBDGWYD6MKTFLXLIAOQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.699968812 (58h11m42.544770338s) [noise=true , address=45.91.116.8:4047 , piece_id=PHW7AI4G2ZUOP7QVQY42QJSIJGT3YYPKU2HPS535LHSFKNHP7UHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.707223719 (58h11m42.55202523s) [noise=true , address=45.91.116.5:4298 , piece_id=IVMB6OERC7RU3RNGOB2ZVYDV62U37SPXVXJCJL7LPJHPS27VBEPA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.708984847 (58h11m42.553786361s) [noise=true , address=45.91.116.9:4263 , piece_id=RBP3ALZHNDFXOOGLURAJRIZ4IHGIFO7PBHQVUEIZU2DLQZYY24UA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.710649854 (58h11m42.555451377s) [noise=true , address=45.91.116.6:4974 , piece_id=K2T65QLUDGAHSH2ICN4IAG5ZFLWSAHQSZ22AFEMHDOGOLO4HMUWA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:46:59.715423554 (58h11m42.56022508s) [noise=true , address=45.91.116.8:4048 , piece_id=Z2HWMDRJGVNY5IK3BQX32QEJBDVFYHNLT3JI6BJRTK2BA2YUEAHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:12.547575970 (58h11m55.392377494s) [noise=true , address=77.90.31.15:4699 , piece_id=22WXQRZAEOM67ZDZCEX3P4HLSRJO7J6PBFTGRQHVEEZYNHNWUZLQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: <nil> Jul 26 21:47:29.253110224 (58h12m12.097911758s) [noise=true , address=104.36.82.8:4970 , piece_id=ZF3NCTXU24JUXXK3YZKLI4BVLGAYD5KK7E3IG5BSPHIVVKGS4JUA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253197097 (58h12m12.097998686s) [noise=true , address=104.36.82.5:4838 , piece_id=2SXVAZUJWOL5XLJOS5WRI2SKDQCWEXCXPXCQCQCAUCNI3ZJHRF4A, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:38942->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253230948 (58h12m12.098032478s) [noise=true , address=104.36.82.10:4078 , piece_id=I4LSWKCK4X2FULIQDBFBA6BXVRBYER4QVL7IH4MIQJEYE3UBWQHQ, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253280622 (58h12m12.098082149s) [noise=true , address=104.36.82.2:4732 , piece_id=OSXQXFS7LBRT5D2TIHP25CZ7I6YMS62QP4IUC5YO7QD5V4V7Y4JA, seg_pos={0 0}, upload=55750]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:47:29.253322073 (58h12m12.098123607s) [seg_pos={0 0}, upload=55750]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:47:29.276472271 (58h12m12.121273798s) [upload=55750]: Done uploading segments... Jul 26 21:47:29.276530550 (58h12m12.121332083s) [upload=55750]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.04515224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X.......X.......X...X.] duration: 11.095463067s FAIL (2025-07-26 21:47:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 11.095482831s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X.......X.......X...X] duration: 30.045643684s FAIL (2025-07-26 21:47:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.045667097s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 1m0.027105666s FAIL (2025-07-26 21:46:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== ) [noise=true , address=104.36.82.55:4221 , piece_id=HBK6VXOOSW24CNXQED22WBVCBBMHDDYX7V65MT2KCGRPB2N5TYVA, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.271471607 (58h10m42.116273144s) [noise=true , address=77.90.31.16:4766 , piece_id=RM7DS33IPBPNFDCMFXYQX4T42YVYKBCMOE45C6KDP44FDANWK43Q, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.271847490 (58h10m42.116649025s) [noise=true , address=104.36.82.49:4002 , piece_id=Z5S6QP3IVCATCETN64QKADIF6UHBHFIOSWIROXJC57JACF2SG5QA, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.271916114 (58h10m42.116717648s) [noise=true , address=104.36.82.13:4217 , piece_id=IWYIFWZANRRCCTWXA7UZ4PLVGZLV32ZINVNBOH36NLEWVR6F5DKQ, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.272380946 (58h10m42.117182478s) [noise=true , address=79.127.205.246:4283 , piece_id=RO2QQBPAUAHAUQBBWXZVEQORE4TTYCN5HBBDCR2DN4HHWIHLJVQA, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.272672777 (58h10m42.117474307s) [noise=true , address=104.36.82.54:4157 , piece_id=WYBOIJLSMYEYSZ6H7EM74OEWEQJMJ7I2BYK5HJ6JAYDUDNVAZHFA, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.272847353 (58h10m42.117648892s) [noise=true , address=104.36.82.52:4110 , piece_id=OEZAGHGQURR7GRSVZA5KA2WF6PZKH4U45KWR6ZDFH6RNYXVQRKEQ, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.273376018 (58h10m42.118177545s) [noise=true , address=104.36.82.53:4150 , piece_id=6FXIPDSICPOIT4F4H4PSSAAK2RPYKVUZF5PY4XHRSOJ56QAACCSQ, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.275468928 (58h10m42.120270461s) [noise=true , address=104.36.82.7:4919 , piece_id=3N6L64LHAZBD3UIZ4JP3FYGA7EWBRQLOX22KHJ6NG24VOAGG7JJA, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.275863389 (58h10m42.120664933s) [noise=true , address=104.36.82.10:4088 , piece_id=5SDKFSNHFQ5UAVWDDHGS5PY4FONKUYYXBVJQ4YFI5CMMCZSWHEQA, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.276326203 (58h10m42.121127747s) [noise=true , address=77.90.31.18:4815 , piece_id=7TKU4C2TJR2W3LOHMYBWER6VGWQ7IRMI2AYDPU7KWSZ6JSCTU6YA, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.276730469 (58h10m42.121532002s) [noise=true , address=104.36.82.47:4521 , piece_id=RKCUNBTM3GV2UDUK6MY6FYBQJD3EHOU575LGPK5JJVFJPRUGPMOQ, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.277171948 (58h10m42.121973479s) [noise=true , address=104.36.82.4:4810 , piece_id=SXWFBPPPX73Q7IEZJ4U7EP47CN4RRP5K2N7RQT7JCPMUQCCOPH6Q, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.279213756 (58h10m42.124015304s) [noise=true , address=104.36.82.20:4452 , piece_id=C7MXG23LUR54UWCKUMJQF3CKOWU7LXDDOZT4W4LQ6FITI7SAFHJQ, seg_pos={0 0}, upload=55735]: Uploading piece... Jul 26 21:45:59.745718845 (58h10m42.590520378s) [upload=55735]: Next returned nil segment Jul 26 21:45:59.745760621 (58h10m42.590562152s) [upload=55735]: Waiting for error group managing segments... Jul 26 21:45:59.927551327 (58h10m42.77235284s) [noise=true , address=79.127.205.246:4283 , piece_id=RO2QQBPAUAHAUQBBWXZVEQORE4TTYCN5HBBDCR2DN4HHWIHLJVQA, seg_pos={0 0}, upload=55735]: Done uploading piece. err: <nil> Jul 26 21:45:59.988085480 (58h10m42.832886996s) [noise=true , address=79.127.205.235:4682 , piece_id=3SE5NLBTFM7XUHUVJGCSTMPCJDSKYIA2DFWLQRVYFNSRSXEZPD7Q, seg_pos={0 0}, upload=55735]: Done uploading piece. err: <nil> Jul 26 21:46:05.038152396 (58h10m47.882953921s) [noise=true , address=77.90.31.18:4815 , piece_id=7TKU4C2TJR2W3LOHMYBWER6VGWQ7IRMI2AYDPU7KWSZ6JSCTU6YA, seg_pos={0 0}, upload=55735]: Done uploading piece. err: <nil> Jul 26 21:46:09.596775297 (58h10m52.441576823s) [noise=true , address=77.90.31.16:4766 , piece_id=RM7DS33IPBPNFDCMFXYQX4T42YVYKBCMOE45C6KDP44FDANWK43Q, seg_pos={0 0}, upload=55735]: Done uploading piece. err: <nil> Jul 26 21:46:59.115854970 (58h11m41.960656492s) [noise=true , address=104.36.82.13:4217 , piece_id=IWYIFWZANRRCCTWXA7UZ4PLVGZLV32ZINVNBOH36NLEWVR6F5DKQ, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:123KLtoTMCQ9DZUjJ8JJab4D5bc7UGTt7yjVV8wDgFTozdZ5gpi, address:104.36.82.13:4217): protocol: rpc: dial tcp 104.36.82.13:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:46:59.115922739 (58h11m41.960724251s) [noise=true , address=104.36.82.52:4110 , piece_id=OEZAGHGQURR7GRSVZA5KA2WF6PZKH4U45KWR6ZDFH6RNYXVQRKEQ, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:46:59.115975968 (58h11m41.96077748s) [noise=true , address=104.36.82.22:4530 , piece_id=I2YTWXZ4NNYTNAW2TEZJPTN3IV4DSTLLU3RIWA4TSQI76OZEPJBQ, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:4530): protocol: rpc: dial tcp 104.36.82.22:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:46:59.116137563 (58h11m41.960939079s) [noise=true , address=104.36.82.56:4254 , piece_id=NKWQZFT6P243MSUT7BCV22DFDT2FPYJHSKZFY5NEJIRCP7Q6TOEA, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:1k4x3HL3PNHwKJX83RyqvnsATHk3HmeMiC12A7raLN58wzvkUN, address:104.36.82.56:4254): protocol: rpc: dial tcp 104.36.82.56:4254: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:46:59.116271645 (58h11m41.961073156s) [noise=true , address=104.36.82.4:4817 , piece_id=WT5IDRACPYYCUKC7ID5Q6HVVUBJ72GH626IZYT6G42KHNYZDD66Q, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz, address:104.36.82.4:4817): protocol: read tcp 159.223.143.39:34102->104.36.82.4:4817: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:46:59.116369827 (58h11m41.961171339s) [noise=true , address=104.36.82.47:4521 , piece_id=RKCUNBTM3GV2UDUK6MY6FYBQJD3EHOU575LGPK5JJVFJPRUGPMOQ, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD, address:104.36.82.47:4521): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:46:59.116459882 (58h11m41.961261396s) [noise=true , address=104.36.82.54:4157 , piece_id=WYBOIJLSMYEYSZ6H7EM74OEWEQJMJ7I2BYK5HJ6JAYDUDNVAZHFA, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:46:59.116535176 (58h11m41.961336688s) [noise=true , address=104.36.82.4:4810 , piece_id=SXWFBPPPX73Q7IEZJ4U7EP47CN4RRP5K2N7RQT7JCPMUQCCOPH6Q, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:12GTWz2fWHLAApibeARUerDvJ9c4AWYRVAvumRxgJwwqCYYJJfg, address:104.36.82.4:4810): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:46:59.116617711 (58h11m41.961419237s) [noise=true , address=104.36.82.10:4088 , piece_id=5SDKFSNHFQ5UAVWDDHGS5PY4FONKUYYXBVJQ4YFI5CMMCZSWHEQA, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT, address:104.36.82.10:4088): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:46:59.116832861 (58h11m41.961634372s) [noise=true , address=104.36.82.55:4221 , piece_id=HBK6VXOOSW24CNXQED22WBVCBBMHDDYX7V65MT2KCGRPB2N5TYVA, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:1ssFiXk5oKTgnVMJ5TBJuSXuD1D2C46gT2hcY6yaDxf8fL1cjm, address:104.36.82.55:4221): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:46:59.116904882 (58h11m41.961706404s) [noise=true , address=104.36.82.49:4002 , piece_id=Z5S6QP3IVCATCETN64QKADIF6UHBHFIOSWIROXJC57JACF2SG5QA, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo, address:104.36.82.49:4002): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:46:59.117022999 (58h11m41.961824511s) [noise=true , address=104.36.82.7:4919 , piece_id=3N6L64LHAZBD3UIZ4JP3FYGA7EWBRQLOX22KHJ6NG24VOAGG7JJA, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:182cMUjmC3Ku6z4KucQpqTS8NcKz7VJKeXt9tFYPnUtDUj6A2w, address:104.36.82.7:4919): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:46:59.117106611 (58h11m41.961908123s) [noise=true , address=104.36.82.53:4150 , piece_id=6FXIPDSICPOIT4F4H4PSSAAK2RPYKVUZF5PY4XHRSOJ56QAACCSQ, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:46:59.117274404 (58h11m41.962075916s) [noise=true , address=104.36.82.20:4452 , piece_id=C7MXG23LUR54UWCKUMJQF3CKOWU7LXDDOZT4W4LQ6FITI7SAFHJQ, seg_pos={0 0}, upload=55735]: Done uploading piece. err: ecclient: upload failed (node:1HX5YAgmkBYJyw6bAY1vz4ov5aBeDsXjU4s7pxpBKAyLbG8yN6, address:104.36.82.20:4452): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:46:59.117296926 (58h11m41.96209845s) [seg_pos={0 0}, upload=55735]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 21:46:59.141014287 (58h11m41.985815802s) [upload=55735]: Done uploading segments... Jul 26 21:46:59.141047265 (58h11m41.985848793s) [upload=55735]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.027125891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 1m0.031034699s FAIL (2025-07-26 21:45:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 5716]: Uploading piece... Jul 26 21:44:59.096744454 (58h9m41.941545983s) [noise=true , address=104.36.82.51:4063 , piece_id=YIYYODAORS5QBXMJBFXWQFTYPYFUWGXOY6N3BBWEGOB6Y674CPYQ, seg_pos={0 0}, upload=55716]: Uploading piece... Jul 26 21:44:59.097020163 (58h9m41.941821695s) [noise=true , address=104.36.82.41:4792 , piece_id=OSG7HWBKKQMMRGGOYSNZXJ7IBMIZZJZYFU7BCYX4SQDCRL5SFLSQ, seg_pos={0 0}, upload=55716]: Uploading piece... Jul 26 21:44:59.097058913 (58h9m41.941860452s) [noise=true , address=104.36.82.55:4217 , piece_id=K3AFF3ABELX4SUR67W42SVWBOQ2N6MH6F3PZHTV3GQZ7FLR3ZZ4A, seg_pos={0 0}, upload=55716]: Uploading piece... Jul 26 21:44:59.097438359 (58h9m41.942239891s) [noise=true , address=64.49.13.169:4716 , piece_id=6NQJHWUESJYCIMWODOSUJBFYCXCR2DXRFSW3XSW4VIREYJT42JMQ, seg_pos={0 0}, upload=55716]: Uploading piece... Jul 26 21:44:59.097453713 (58h9m41.942255247s) [noise=true , address=104.36.82.50:4014 , piece_id=VPQ5XQDYAFWIKBQLOKREJNKNICFZC7G4SBXA3DWFSCUZDFI5H4JQ, seg_pos={0 0}, upload=55716]: Uploading piece... Jul 26 21:44:59.097817599 (58h9m41.942619131s) [noise=true , address=104.36.82.10:4081 , piece_id=OLCIDTD2QSP5X4KVPJUBUJR7PUZTY22JNKYAZDTFNCBYXAM3GDLA, seg_pos={0 0}, upload=55716]: Uploading piece... Jul 26 21:44:59.097888146 (58h9m41.942689677s) [noise=true , address=64.49.13.137:4500 , piece_id=7PFEHHPALWMVI3GR4GJQOSEDFXAKYLAVYO6T7QOLFBCURY7G2XXQ, seg_pos={0 0}, upload=55716]: Uploading piece... Jul 26 21:44:59.098185093 (58h9m41.942986622s) [noise=true , address=104.36.82.54:4177 , piece_id=PFLOK34SF3PPACH2DU2C6XNJMFTEJD43KXIUSKPFRRU4BPP3XP6A, seg_pos={0 0}, upload=55716]: Uploading piece... Jul 26 21:44:59.098316856 (58h9m41.943118398s) [noise=true , address=79.127.205.235:4685 , piece_id=XKT3GEZZW6DK4HMYEF2BYXTZSBLSL6GQN5MP673UCB6IPDHH7GCQ, seg_pos={0 0}, upload=55716]: Uploading piece... Jul 26 21:44:59.103231614 (58h9m41.948033148s) [noise=true , address=104.36.82.53:4156 , piece_id=Q2TECQKETI34Q4FXUJAFYZGADV5CXQBZ7OZQYLHKUOZUJ4VPTIMA, seg_pos={0 0}, upload=55716]: Uploading piece... Jul 26 21:44:59.106875013 (58h9m41.951676541s) [noise=true , address=104.36.82.4:4811 , piece_id=2JRLEBQERZAJZZSXX23YZC2F5PRHXS3DOD3BEZAWF3WXPVXWPPIA, seg_pos={0 0}, upload=55716]: Uploading piece... Jul 26 21:44:59.322508703 (58h9m42.167310213s) [upload=55716]: Next returned nil segment Jul 26 21:44:59.322528865 (58h9m42.167330396s) [upload=55716]: Waiting for error group managing segments... Jul 26 21:44:59.393222635 (58h9m42.238024163s) [noise=true , address=79.127.201.215:4358 , piece_id=RJOHUSQXMUSC76K2FNGTO5TGMEOSOXTZW4BFJE434S3XNOGOB22A, seg_pos={0 0}, upload=55716]: Done uploading piece. err: <nil> Jul 26 21:44:59.743970922 (58h9m42.588772434s) [noise=true , address=79.127.205.235:4685 , piece_id=XKT3GEZZW6DK4HMYEF2BYXTZSBLSL6GQN5MP673UCB6IPDHH7GCQ, seg_pos={0 0}, upload=55716]: Done uploading piece. err: <nil> Jul 26 21:44:59.753002393 (58h9m42.597803918s) [noise=true , address=64.49.13.137:4500 , piece_id=7PFEHHPALWMVI3GR4GJQOSEDFXAKYLAVYO6T7QOLFBCURY7G2XXQ, seg_pos={0 0}, upload=55716]: Done uploading piece. err: <nil> Jul 26 21:44:59.803968126 (58h9m42.648769639s) [noise=true , address=64.49.13.169:4716 , piece_id=6NQJHWUESJYCIMWODOSUJBFYCXCR2DXRFSW3XSW4VIREYJT42JMQ, seg_pos={0 0}, upload=55716]: Done uploading piece. err: <nil> Jul 26 21:45:55.665764078 (58h10m38.510565602s) [noise=true , address=104.36.82.4:4811 , piece_id=2JRLEBQERZAJZZSXX23YZC2F5PRHXS3DOD3BEZAWF3WXPVXWPPIA, seg_pos={0 0}, upload=55716]: Done uploading piece. err: <nil> Jul 26 21:45:59.027155873 (58h10m41.871957404s) [noise=true , address=104.36.82.51:4063 , piece_id=YIYYODAORS5QBXMJBFXWQFTYPYFUWGXOY6N3BBWEGOB6Y674CPYQ, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D, address:104.36.82.51:4063): protocol: write tcp 159.223.143.39:34848->104.36.82.51:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.027262263 (58h10m41.872063775s) [noise=true , address=104.36.82.54:4177 , piece_id=PFLOK34SF3PPACH2DU2C6XNJMFTEJD43KXIUSKPFRRU4BPP3XP6A, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 159.223.143.39:44064->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.027343509 (58h10m41.872145036s) [noise=true , address=104.36.82.6:4890 , piece_id=Q5YVQ5WYH2R3BN47PO6KCBC5NUNSW5J4WDSO2YFUJOCLFBHLND2Q, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:1gakF2Cctzc6EhFScfLfBHJdcDHqMBcey4mNoToEwbtk8Pwix5, address:104.36.82.6:4890): protocol: write tcp 159.223.143.39:34102->104.36.82.6:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.027587544 (58h10m41.872389075s) [noise=true , address=104.36.82.12:4181 , piece_id=PGCWRGURJMLTQC7VWR53LP43UJLVWPHOPW7FWVE5RK4FRB74A2CA, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:1XH92Sdi4ZiRg1iic5A9Mp8RyKZsk9WfVQHLoQ8t9Vvk2Lyxwh, address:104.36.82.12:4181): protocol: write tcp 159.223.143.39:45778->104.36.82.12:4181: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.027718312 (58h10m41.872519843s) [noise=true , address=104.36.82.53:4156 , piece_id=Q2TECQKETI34Q4FXUJAFYZGADV5CXQBZ7OZQYLHKUOZUJ4VPTIMA, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: rpc: dial tcp 104.36.82.53:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.027799625 (58h10m41.872601156s) [noise=true , address=104.36.82.52:4092 , piece_id=RZGQO2R2DKNQ6T44IJZ67PCG4XF6V2PFSYRH4ZYVL3JFCTT7ABJQ, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: write tcp 159.223.143.39:34150->104.36.82.52:4092: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.027877092 (58h10m41.872678616s) [noise=true , address=104.36.82.13:4196 , piece_id=7YFCK37PEOKJI2MTLDIXVYLEVCFHDVCCHAV743Z4JLG3G2P3ZEBA, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:1aLZa6kv5VhDgqvUZp2HkzpkbA81KU13iGTTsiVfSeUDzLu8Lu, address:104.36.82.13:4196): protocol: rpc: dial tcp 104.36.82.13:4196: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.027947763 (58h10m41.872749273s) [noise=true , address=104.36.82.49:4011 , piece_id=CZQO4CUUIGPEES6WWKJK55XJCUQ5JL3IGOF6MNXV4RHVYG6PDU3A, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:1DV1WUL3dGBjH23a6xtetmqmLE9fER1FQf53gmYFxZiP25ra8N, address:104.36.82.49:4011): protocol: write tcp 159.223.143.39:57620->104.36.82.49:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.028016974 (58h10m41.872818484s) [noise=true , address=104.36.82.41:4792 , piece_id=OSG7HWBKKQMMRGGOYSNZXJ7IBMIZZJZYFU7BCYX4SQDCRL5SFLSQ, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: rpc: dial tcp 104.36.82.41:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.028093320 (58h10m41.872894836s) [noise=true , address=104.36.82.55:4217 , piece_id=K3AFF3ABELX4SUR67W42SVWBOQ2N6MH6F3PZHTV3GQZ7FLR3ZZ4A, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:12bw9V8UMi7pFxqJ8fh22x5ZPmtJUJyJGw8hD9NhQcfHdfx4cZj, address:104.36.82.55:4217): protocol: write tcp 159.223.143.39:55806->104.36.82.55:4217: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.028209597 (58h10m41.873011146s) [noise=true , address=104.36.82.47:4552 , piece_id=KLRQQWSVSR5LNX5ZGEY62GCKSROLO7E3EXJMW2UTIJRK3RL7KDNA, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.028222733 (58h10m41.873024242s) [noise=true , address=104.36.82.10:4081 , piece_id=OLCIDTD2QSP5X4KVPJUBUJR7PUZTY22JNKYAZDTFNCBYXAM3GDLA, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:12t9rGfCQ7bLkNnZzaPVgkE2Y9rXsiFp1XGdeECzv9M1jk8Qqmp, address:104.36.82.10:4081): protocol: write tcp 159.223.143.39:36444->104.36.82.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.028260306 (58h10m41.873061836s) [noise=true , address=104.36.82.50:4014 , piece_id=VPQ5XQDYAFWIKBQLOKREJNKNICFZC7G4SBXA3DWFSCUZDFI5H4JQ, seg_pos={0 0}, upload=55716]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:104.36.82.50:4014): protocol: rpc: dial tcp 104.36.82.50:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:45:59.028339107 (58h10m41.87314064s) [seg_pos={0 0}, upload=55716]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 21:45:59.054225718 (58h10m41.89902723s) [upload=55716]: Done uploading segments... Jul 26 21:45:59.054256017 (58h10m41.899057544s) [upload=55716]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.031051682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................X.XX...X.X...] duration: 5.027221377s FAIL (2025-07-26 21:45:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=55680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: rpc: dial tcp 104.36.82.2:4732: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:47.771002362 (58h7m30.615803892s) [seg_pos={0 0}, upload=55680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:42:47.771184279 (58h7m30.615985833s) [upload=55680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:42:47.771192227 (58h7m30.615993753s) [upload=55680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:42:47.814292768 (58h7m30.659094298s) [upload=55680]: Done uploading segments... Jul 26 21:42:47.814316231 (58h7m30.65911776s) [upload=55680]: Upload finished. err: <nil> Jul 26 21:43:39.374613126 (58h8m22.219414639s) [upload=55694]: Starting upload Jul 26 21:43:39.374680877 (58h8m22.219482401s) [upload=55694]: Done starting upload Jul 26 21:43:39.374730849 (58h8m22.219532359s) [upload=55694]: Uploading segments... Jul 26 21:43:39.374965546 (58h8m22.219767058s) [upload=55694]: Got next segment. Inline: false Jul 26 21:43:39.375010672 (58h8m22.219812183s) [upload=55694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:43:39.375035700 (58h8m22.219837223s) [upload=55694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:43:39.400062014 (58h8m22.244863545s) [seg_pos={0 0}, upload=55694]: Begin upload segment... Jul 26 21:43:39.400128495 (58h8m22.244930025s) [seg_pos={0 0}, upload=55694]: Done begin upload segment. Jul 26 21:43:39.400297010 (58h8m22.245098546s) [noise=true , address=104.36.82.2:4729 , piece_id=2EW64YCU54XTYJSCKNHEZ7OTEGPNSMHPV4Q3Z3BUH5IUSV7WVOVA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.401011243 (58h8m22.245812775s) [noise=true , address=45.91.116.9:4261 , piece_id=NTI5BHSVNRQA3XEOSBDEFVCIIVL6JYSJMU7TBKRYGCHH3ORNMTOA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.401089603 (58h8m22.245891141s) [noise=true , address=104.36.82.6:4875 , piece_id=R2U7D4HYLWFRUYHOTYEYQATGGJAYAFB55V6XAWRVTMFR5DGXV23Q, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.402178452 (58h8m22.246979977s) [noise=true , address=45.91.116.8:4047 , piece_id=67P2ZZO3CI6WA5LUDE3GGF4F7DEEONZQ7X6FK5RXGB5FC3NPRNTA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.402248273 (58h8m22.247049795s) [noise=true , address=45.91.116.12:4227 , piece_id=EUJ6RI2H2P4AS44LKGLR7BUZ4BH3THAFZKACRRXAAI3HVWB6JSHA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403113183 (58h8m22.247914716s) [noise=true , address=77.90.31.17:4770 , piece_id=4VG4ZRU3YI7UTL4SGTWUOMHV6R7TXWNSS5LSVPDC5LOWCSOMO53A, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403289478 (58h8m22.248091001s) [noise=true , address=45.91.116.13:4155 , piece_id=OAYNJLFC2VOQTZLPGOV57SN7KPBJGCBIHH7RLN4OE5WFX345ISIA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403971148 (58h8m22.24877267s) [noise=true , address=104.36.82.12:4151 , piece_id=XAQ4DZ3B7YRQR2R5Z4TKZ65AGIKPKTJBMGZH6TDVWIVJAH5MRHEA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.404780442 (58h8m22.249581965s) [noise=true , address=45.91.116.11:4117 , piece_id=K267FD7VGIV6QOUP6A42KD3NHWY5LSOGBHK7D5BDREVHLLWH6QBA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.406562224 (58h8m22.251363756s) [noise=true , address=45.91.116.13:4153 , piece_id=FA5EMKYH7FSDYSQ4ZMUDB55G2OP4HQAFS7EATB6FHOIJIFQEQLJQ, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.408084243 (58h8m22.252885758s) [noise=true , address=77.90.31.18:4805 , piece_id=UZROHZAWFFXDODGQ4BQ3AVOBL3XVOT22D7FBSTQTUGIUAJJ4O56A, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.414472700 (58h8m22.259274213s) [noise=true , address=45.91.116.5:4297 , piece_id=UFXGLSYAJHN3WQR42P5JJ43YK5C76SX2LEHWB5DJXEK5MFNRRTTA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.417180066 (58h8m22.261981582s) [noise=true , address=104.36.82.3:4765 , piece_id=GP2C2PTQ6TYUZE2QKEK7ZVMZJLSWAV3NLOTOEAKUVGUHCQYB57OQ, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.419304498 (58h8m22.264106014s) [noise=true , address=45.91.116.14:4191 , piece_id=J2GRG7KNFB5ONW3A7MG5F6ZBC2GUARYD4PDOASHALUCIGMQJ3PMA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.421292752 (58h8m22.266094267s) [noise=true , address=45.91.116.14:4190 , piece_id=WLAJRLI2ZY5J4SFQZT5PEQTJBFZVVUVXBHG7ERGBJ5FWDAFOI2SA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.423400915 (58h8m22.268202424s) [noise=true , address=45.91.116.8:4045 , piece_id=BNWRAYUXYSSPC57U3U526OJWOXLBWUMHMLFUBNW5HVMFY4E5MRRA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.493673489 (58h8m22.338475015s) [upload=55694]: Next returned nil segment Jul 26 21:43:39.493698457 (58h8m22.33849999s) [upload=55694]: Waiting for error group managing segments... Jul 26 21:43:39.497547530 (58h8m22.342349039s) [noise=true , address=45.91.116.8:4045 , piece_id=BNWRAYUXYSSPC57U3U526OJWOXLBWUMHMLFUBNW5HVMFY4E5MRRA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.498138143 (58h8m22.342939667s) [noise=true , address=45.91.116.5:4297 , piece_id=UFXGLSYAJHN3WQR42P5JJ43YK5C76SX2LEHWB5DJXEK5MFNRRTTA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.498607426 (58h8m22.343408935s) [noise=true , address=45.91.116.13:4153 , piece_id=FA5EMKYH7FSDYSQ4ZMUDB55G2OP4HQAFS7EATB6FHOIJIFQEQLJQ, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499012857 (58h8m22.343814366s) [noise=true , address=45.91.116.9:4261 , piece_id=NTI5BHSVNRQA3XEOSBDEFVCIIVL6JYSJMU7TBKRYGCHH3ORNMTOA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499439665 (58h8m22.344241174s) [noise=true , address=45.91.116.13:4155 , piece_id=OAYNJLFC2VOQTZLPGOV57SN7KPBJGCBIHH7RLN4OE5WFX345ISIA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499977307 (58h8m22.34477883s) [noise=true , address=45.91.116.14:4190 , piece_id=WLAJRLI2ZY5J4SFQZT5PEQTJBFZVVUVXBHG7ERGBJ5FWDAFOI2SA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.500549087 (58h8m22.345350624s) [noise=true , address=45.91.116.11:4117 , piece_id=K267FD7VGIV6QOUP6A42KD3NHWY5LSOGBHK7D5BDREVHLLWH6QBA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501023407 (58h8m22.345824929s) [noise=true , address=45.91.116.8:4047 , piece_id=67P2ZZO3CI6WA5LUDE3GGF4F7DEEONZQ7X6FK5RXGB5FC3NPRNTA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501481940 (58h8m22.346283472s) [noise=true , address=45.91.116.12:4227 , piece_id=EUJ6RI2H2P4AS44LKGLR7BUZ4BH3THAFZKACRRXAAI3HVWB6JSHA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501986448 (58h8m22.346787991s) [noise=true , address=45.91.116.14:4191 , piece_id=J2GRG7KNFB5ONW3A7MG5F6ZBC2GUARYD4PDOASHALUCIGMQJ3PMA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:43.671096078 (58h8m26.515897595s) [noise=true , address=77.90.31.17:4770 , piece_id=4VG4ZRU3YI7UTL4SGTWUOMHV6R7TXWNSS5LSVPDC5LOWCSOMO53A, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:49.036287942 (58h8m31.881089454s) [noise=true , address=77.90.31.18:4805 , piece_id=UZROHZAWFFXDODGQ4BQ3AVOBL3XVOT22D7FBSTQTUGIUAJJ4O56A, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:44:09.341718713 (58h8m52.186520228s) [noise=true , address=104.36.82.12:4151 , piece_id=XAQ4DZ3B7YRQR2R5Z4TKZ65AGIKPKTJBMGZH6TDVWIVJAH5MRHEA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:47272->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341791849 (58h8m52.186593361s) [noise=true , address=104.36.82.6:4875 , piece_id=R2U7D4HYLWFRUYHOTYEYQATGGJAYAFB55V6XAWRVTMFR5DGXV23Q, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:50054->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341847201 (58h8m52.186648712s) [noise=true , address=104.36.82.2:4729 , piece_id=2EW64YCU54XTYJSCKNHEZ7OTEGPNSMHPV4Q3Z3BUH5IUSV7WVOVA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:51548->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341864106 (58h8m52.186665628s) [noise=true , address=104.36.82.3:4765 , piece_id=GP2C2PTQ6TYUZE2QKEK7ZVMZJLSWAV3NLOTOEAKUVGUHCQYB57OQ, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:47384->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341909603 (58h8m52.186711143s) [seg_pos={0 0}, upload=55694]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:44:09.368838649 (58h8m52.213640171s) [upload=55694]: Done uploading segments... Jul 26 21:44:09.368863899 (58h8m52.213665413s) [upload=55694]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.027252303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.......X.......X...] duration: 3.14442599s FAIL (2025-07-26 21:45:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.144451421s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X.......X.......X..] duration: 7.98538825s FAIL (2025-07-26 21:45:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.985419001s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 1m0.036499243s FAIL (2025-07-26 21:44:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 2CELKRSXEHIV5PLPKYXKX2TBRXKYFHLZVOIMDFQ, seg_pos={0 0}, upload=55701]: Uploading piece... Jul 26 21:43:59.013246872 (58h8m41.858048399s) [noise=true , address=104.36.82.52:4092 , piece_id=WROMC4V66LO4KREPOD2XUNHOF43BAYWWS4AS3AKXROQEKFNO5KEQ, seg_pos={0 0}, upload=55701]: Uploading piece... Jul 26 21:43:59.013635412 (58h8m41.858436937s) [noise=true , address=45.91.116.14:4206 , piece_id=2S3UWEVVAKOCLLS7XJVIHEMB4FIROF7ZP2KCGUYK66MMVY2YEGSQ, seg_pos={0 0}, upload=55701]: Uploading piece... Jul 26 21:43:59.013652745 (58h8m41.858454278s) [noise=true , address=104.36.82.54:4181 , piece_id=L22NKOK24JV5K5IRLDJ44SOWXR5O7PVWKG5OEUAD7FCD5366BVWQ, seg_pos={0 0}, upload=55701]: Uploading piece... Jul 26 21:43:59.014016991 (58h8m41.858818521s) [noise=true , address=79.127.201.215:4350 , piece_id=C5HHJB5SL7DTHBK6PBKA35HBBEFU2N2XQM6KBPNO4IV434JGA6IA, seg_pos={0 0}, upload=55701]: Uploading piece... Jul 26 21:43:59.014041299 (58h8m41.858842832s) [noise=true , address=79.127.205.235:4684 , piece_id=T3TPYCWHKN7ULLBUBWI5WKXNF76QV6BSQ2KD4VJJ2AMT44JZ362A, seg_pos={0 0}, upload=55701]: Uploading piece... Jul 26 21:43:59.014407576 (58h8m41.859209107s) [noise=true , address=104.36.82.53:4139 , piece_id=B3WA7A6M5OY3FVHVHVIMHJBLTYBHP5LSYJXVU2FOEXXJW2Z374CA, seg_pos={0 0}, upload=55701]: Uploading piece... Jul 26 21:43:59.014432304 (58h8m41.859233839s) [noise=true , address=104.36.82.40:4971 , piece_id=MDQWQNIMS3AITV6BN5VNOEOS433ZGUYBSTC7EI7ZVO7TAAQMXJWQ, seg_pos={0 0}, upload=55701]: Uploading piece... Jul 26 21:43:59.014764287 (58h8m41.859565812s) [noise=true , address=104.36.82.46:4508 , piece_id=RJUARQBENR45MS6GF7JEAJ37X4XWMBKW7JUKFDA4KTNJ3ZIF3WTA, seg_pos={0 0}, upload=55701]: Uploading piece... Jul 26 21:43:59.014821814 (58h8m41.859623336s) [noise=true , address=104.36.82.11:4140 , piece_id=EHCPZ5HEKLYF56QOQC3CA5JRJLYF2T4Z56H42EX22SJDZIEW4IUQ, seg_pos={0 0}, upload=55701]: Uploading piece... Jul 26 21:43:59.015135694 (58h8m41.859937222s) [noise=true , address=104.36.82.49:4978 , piece_id=HI2BEZ4477V3TARKBKWIEKKHAVT3BTHW2U3BYBWN6ME4EBILRZHQ, seg_pos={0 0}, upload=55701]: Uploading piece... Jul 26 21:43:59.089907225 (58h8m41.934708754s) [upload=55701]: Next returned nil segment Jul 26 21:43:59.089928617 (58h8m41.934730149s) [upload=55701]: Waiting for error group managing segments... Jul 26 21:43:59.096272170 (58h8m41.941073679s) [noise=true , address=79.127.219.51:4210 , piece_id=YFEQNVRBWYNCQOPEHGCUG55G6PY7NZYQPXVKXZ4NTN2AHUNG6RDQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: <nil> Jul 26 21:44:00.132225026 (58h8m42.977026552s) [noise=true , address=79.127.205.235:4684 , piece_id=T3TPYCWHKN7ULLBUBWI5WKXNF76QV6BSQ2KD4VJJ2AMT44JZ362A, seg_pos={0 0}, upload=55701]: Done uploading piece. err: <nil> Jul 26 21:44:02.515334060 (58h8m45.360135572s) [noise=true , address=77.90.31.18:4821 , piece_id=VTXOSOG766NPWSAADNPXX4SPE4DTWHVKX45PGVDWTLX2XTOBCIRQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: <nil> Jul 26 21:44:02.593918694 (58h8m45.438720238s) [noise=true , address=45.91.116.14:4206 , piece_id=2S3UWEVVAKOCLLS7XJVIHEMB4FIROF7ZP2KCGUYK66MMVY2YEGSQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: <nil> Jul 26 21:44:03.001905710 (58h8m45.846707259s) [noise=true , address=79.127.201.215:4350 , piece_id=C5HHJB5SL7DTHBK6PBKA35HBBEFU2N2XQM6KBPNO4IV434JGA6IA, seg_pos={0 0}, upload=55701]: Done uploading piece. err: <nil> Jul 26 21:44:58.940668507 (58h9m41.785470033s) [noise=true , address=104.36.82.54:4181 , piece_id=L22NKOK24JV5K5IRLDJ44SOWXR5O7PVWKG5OEUAD7FCD5366BVWQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:12gGf5E7tzF2TXtLWgpK55t24SC1BpTSeC9G66nPmR1ARnDrAE8, address:104.36.82.54:4181): protocol: write tcp 159.223.143.39:58770->104.36.82.54:4181: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.940733923 (58h9m41.785535445s) [noise=true , address=104.36.82.4:4813 , piece_id=7WN2KWYS7J3IK64RCGMS3JE6U6LDH6RKXO2J5Z2O6AC45EQCVDUQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:104.36.82.4:4813): protocol: write tcp 159.223.143.39:41202->104.36.82.4:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.940784573 (58h9m41.785586084s) [noise=true , address=104.36.82.42:4797 , piece_id=ZZA36AAMT3XZ32CELKRSXEHIV5PLPKYXKX2TBRXKYFHLZVOIMDFQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:104.36.82.42:4797): protocol: write tcp 159.223.143.39:59348->104.36.82.42:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.940836963 (58h9m41.785638485s) [noise=true , address=104.36.82.52:4092 , piece_id=WROMC4V66LO4KREPOD2XUNHOF43BAYWWS4AS3AKXROQEKFNO5KEQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: write tcp 159.223.143.39:45256->104.36.82.52:4092: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.940893580 (58h9m41.785695092s) [noise=true , address=104.36.82.40:4971 , piece_id=MDQWQNIMS3AITV6BN5VNOEOS433ZGUYBSTC7EI7ZVO7TAAQMXJWQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:1pYDfLBXpGNfJsH2sNUSSuKPTtgXNUEZx1oGB4SxzsFwdX4UiH, address:104.36.82.40:4971): protocol: write tcp 159.223.143.39:35900->104.36.82.40:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.940950929 (58h9m41.785752442s) [noise=true , address=104.36.82.46:4508 , piece_id=RJUARQBENR45MS6GF7JEAJ37X4XWMBKW7JUKFDA4KTNJ3ZIF3WTA, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: write tcp 159.223.143.39:38640->104.36.82.46:4508: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.940993698 (58h9m41.785795247s) [noise=true , address=104.36.82.56:4249 , piece_id=4OX4OZG4KO7OLLQ3JDDW2NMKWJGKA76BW4XTPNMSL5T6DCEELPEQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: rpc: dial tcp 104.36.82.56:4249: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.941025152 (58h9m41.785826676s) [noise=true , address=104.36.82.47:4533 , piece_id=CCOZWXE2SBD5BLWIH52XSTFNNWTDWFL2WVZN576Z4YGUY5IA74WA, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: write tcp 159.223.143.39:35948->104.36.82.47:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.941058058 (58h9m41.78585958s) [noise=true , address=104.36.82.11:4140 , piece_id=EHCPZ5HEKLYF56QOQC3CA5JRJLYF2T4Z56H42EX22SJDZIEW4IUQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:12AYQRkKcRprBgX1BdUwnd4MCQ1swYucsG9aYvLLEaskgrngkCB, address:104.36.82.11:4140): protocol: write tcp 159.223.143.39:38242->104.36.82.11:4140: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.941081617 (58h9m41.785883143s) [noise=true , address=104.36.82.53:4139 , piece_id=B3WA7A6M5OY3FVHVHVIMHJBLTYBHP5LSYJXVU2FOEXXJW2Z374CA, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: write tcp 159.223.143.39:57432->104.36.82.53:4139: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.941139672 (58h9m41.785941183s) [noise=true , address=104.36.82.49:4978 , piece_id=HI2BEZ4477V3TARKBKWIEKKHAVT3BTHW2U3BYBWN6ME4EBILRZHQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:104.36.82.49:4978): protocol: write tcp 159.223.143.39:44846->104.36.82.49:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.941166141 (58h9m41.785967663s) [noise=true , address=104.36.82.14:4242 , piece_id=D7H4OCC3XOIIGD33Y5JPU4UD7JUWGALK2HLF74RKKLW6D5HLFWRQ, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz, address:104.36.82.14:4242): protocol: write tcp 159.223.143.39:49566->104.36.82.14:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.941221796 (58h9m41.78602332s) [noise=true , address=104.36.82.43:4861 , piece_id=RI57EZJCDMRC2CC7Z3QO3W2UYHNI7JSZXUROIJI4N4GTY6MHPA7Q, seg_pos={0 0}, upload=55701]: Done uploading piece. err: ecclient: upload failed (node:122HC6BetMaBDNGEaHVwmadWtV1nCHx3vGBK2SWVDEQS1iBCz4o, address:104.36.82.43:4861): protocol: write tcp 159.223.143.39:36856->104.36.82.43:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:58.941252800 (58h9m41.786054323s) [seg_pos={0 0}, upload=55701]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 21:44:58.974790159 (58h9m41.819591671s) [upload=55701]: Done uploading segments... Jul 26 21:44:58.974828198 (58h9m41.819629729s) [upload=55701]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.036529269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................X.XX...X.X..] duration: 3.041426174s FAIL (2025-07-26 21:44:52 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=55680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: rpc: dial tcp 104.36.82.2:4732: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:47.771002362 (58h7m30.615803892s) [seg_pos={0 0}, upload=55680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:42:47.771184279 (58h7m30.615985833s) [upload=55680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:42:47.771192227 (58h7m30.615993753s) [upload=55680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:42:47.814292768 (58h7m30.659094298s) [upload=55680]: Done uploading segments... Jul 26 21:42:47.814316231 (58h7m30.65911776s) [upload=55680]: Upload finished. err: <nil> Jul 26 21:43:39.374613126 (58h8m22.219414639s) [upload=55694]: Starting upload Jul 26 21:43:39.374680877 (58h8m22.219482401s) [upload=55694]: Done starting upload Jul 26 21:43:39.374730849 (58h8m22.219532359s) [upload=55694]: Uploading segments... Jul 26 21:43:39.374965546 (58h8m22.219767058s) [upload=55694]: Got next segment. Inline: false Jul 26 21:43:39.375010672 (58h8m22.219812183s) [upload=55694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:43:39.375035700 (58h8m22.219837223s) [upload=55694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:43:39.400062014 (58h8m22.244863545s) [seg_pos={0 0}, upload=55694]: Begin upload segment... Jul 26 21:43:39.400128495 (58h8m22.244930025s) [seg_pos={0 0}, upload=55694]: Done begin upload segment. Jul 26 21:43:39.400297010 (58h8m22.245098546s) [noise=true , address=104.36.82.2:4729 , piece_id=2EW64YCU54XTYJSCKNHEZ7OTEGPNSMHPV4Q3Z3BUH5IUSV7WVOVA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.401011243 (58h8m22.245812775s) [noise=true , address=45.91.116.9:4261 , piece_id=NTI5BHSVNRQA3XEOSBDEFVCIIVL6JYSJMU7TBKRYGCHH3ORNMTOA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.401089603 (58h8m22.245891141s) [noise=true , address=104.36.82.6:4875 , piece_id=R2U7D4HYLWFRUYHOTYEYQATGGJAYAFB55V6XAWRVTMFR5DGXV23Q, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.402178452 (58h8m22.246979977s) [noise=true , address=45.91.116.8:4047 , piece_id=67P2ZZO3CI6WA5LUDE3GGF4F7DEEONZQ7X6FK5RXGB5FC3NPRNTA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.402248273 (58h8m22.247049795s) [noise=true , address=45.91.116.12:4227 , piece_id=EUJ6RI2H2P4AS44LKGLR7BUZ4BH3THAFZKACRRXAAI3HVWB6JSHA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403113183 (58h8m22.247914716s) [noise=true , address=77.90.31.17:4770 , piece_id=4VG4ZRU3YI7UTL4SGTWUOMHV6R7TXWNSS5LSVPDC5LOWCSOMO53A, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403289478 (58h8m22.248091001s) [noise=true , address=45.91.116.13:4155 , piece_id=OAYNJLFC2VOQTZLPGOV57SN7KPBJGCBIHH7RLN4OE5WFX345ISIA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403971148 (58h8m22.24877267s) [noise=true , address=104.36.82.12:4151 , piece_id=XAQ4DZ3B7YRQR2R5Z4TKZ65AGIKPKTJBMGZH6TDVWIVJAH5MRHEA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.404780442 (58h8m22.249581965s) [noise=true , address=45.91.116.11:4117 , piece_id=K267FD7VGIV6QOUP6A42KD3NHWY5LSOGBHK7D5BDREVHLLWH6QBA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.406562224 (58h8m22.251363756s) [noise=true , address=45.91.116.13:4153 , piece_id=FA5EMKYH7FSDYSQ4ZMUDB55G2OP4HQAFS7EATB6FHOIJIFQEQLJQ, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.408084243 (58h8m22.252885758s) [noise=true , address=77.90.31.18:4805 , piece_id=UZROHZAWFFXDODGQ4BQ3AVOBL3XVOT22D7FBSTQTUGIUAJJ4O56A, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.414472700 (58h8m22.259274213s) [noise=true , address=45.91.116.5:4297 , piece_id=UFXGLSYAJHN3WQR42P5JJ43YK5C76SX2LEHWB5DJXEK5MFNRRTTA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.417180066 (58h8m22.261981582s) [noise=true , address=104.36.82.3:4765 , piece_id=GP2C2PTQ6TYUZE2QKEK7ZVMZJLSWAV3NLOTOEAKUVGUHCQYB57OQ, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.419304498 (58h8m22.264106014s) [noise=true , address=45.91.116.14:4191 , piece_id=J2GRG7KNFB5ONW3A7MG5F6ZBC2GUARYD4PDOASHALUCIGMQJ3PMA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.421292752 (58h8m22.266094267s) [noise=true , address=45.91.116.14:4190 , piece_id=WLAJRLI2ZY5J4SFQZT5PEQTJBFZVVUVXBHG7ERGBJ5FWDAFOI2SA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.423400915 (58h8m22.268202424s) [noise=true , address=45.91.116.8:4045 , piece_id=BNWRAYUXYSSPC57U3U526OJWOXLBWUMHMLFUBNW5HVMFY4E5MRRA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.493673489 (58h8m22.338475015s) [upload=55694]: Next returned nil segment Jul 26 21:43:39.493698457 (58h8m22.33849999s) [upload=55694]: Waiting for error group managing segments... Jul 26 21:43:39.497547530 (58h8m22.342349039s) [noise=true , address=45.91.116.8:4045 , piece_id=BNWRAYUXYSSPC57U3U526OJWOXLBWUMHMLFUBNW5HVMFY4E5MRRA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.498138143 (58h8m22.342939667s) [noise=true , address=45.91.116.5:4297 , piece_id=UFXGLSYAJHN3WQR42P5JJ43YK5C76SX2LEHWB5DJXEK5MFNRRTTA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.498607426 (58h8m22.343408935s) [noise=true , address=45.91.116.13:4153 , piece_id=FA5EMKYH7FSDYSQ4ZMUDB55G2OP4HQAFS7EATB6FHOIJIFQEQLJQ, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499012857 (58h8m22.343814366s) [noise=true , address=45.91.116.9:4261 , piece_id=NTI5BHSVNRQA3XEOSBDEFVCIIVL6JYSJMU7TBKRYGCHH3ORNMTOA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499439665 (58h8m22.344241174s) [noise=true , address=45.91.116.13:4155 , piece_id=OAYNJLFC2VOQTZLPGOV57SN7KPBJGCBIHH7RLN4OE5WFX345ISIA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499977307 (58h8m22.34477883s) [noise=true , address=45.91.116.14:4190 , piece_id=WLAJRLI2ZY5J4SFQZT5PEQTJBFZVVUVXBHG7ERGBJ5FWDAFOI2SA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.500549087 (58h8m22.345350624s) [noise=true , address=45.91.116.11:4117 , piece_id=K267FD7VGIV6QOUP6A42KD3NHWY5LSOGBHK7D5BDREVHLLWH6QBA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501023407 (58h8m22.345824929s) [noise=true , address=45.91.116.8:4047 , piece_id=67P2ZZO3CI6WA5LUDE3GGF4F7DEEONZQ7X6FK5RXGB5FC3NPRNTA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501481940 (58h8m22.346283472s) [noise=true , address=45.91.116.12:4227 , piece_id=EUJ6RI2H2P4AS44LKGLR7BUZ4BH3THAFZKACRRXAAI3HVWB6JSHA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501986448 (58h8m22.346787991s) [noise=true , address=45.91.116.14:4191 , piece_id=J2GRG7KNFB5ONW3A7MG5F6ZBC2GUARYD4PDOASHALUCIGMQJ3PMA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:43.671096078 (58h8m26.515897595s) [noise=true , address=77.90.31.17:4770 , piece_id=4VG4ZRU3YI7UTL4SGTWUOMHV6R7TXWNSS5LSVPDC5LOWCSOMO53A, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:49.036287942 (58h8m31.881089454s) [noise=true , address=77.90.31.18:4805 , piece_id=UZROHZAWFFXDODGQ4BQ3AVOBL3XVOT22D7FBSTQTUGIUAJJ4O56A, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:44:09.341718713 (58h8m52.186520228s) [noise=true , address=104.36.82.12:4151 , piece_id=XAQ4DZ3B7YRQR2R5Z4TKZ65AGIKPKTJBMGZH6TDVWIVJAH5MRHEA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:47272->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341791849 (58h8m52.186593361s) [noise=true , address=104.36.82.6:4875 , piece_id=R2U7D4HYLWFRUYHOTYEYQATGGJAYAFB55V6XAWRVTMFR5DGXV23Q, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:50054->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341847201 (58h8m52.186648712s) [noise=true , address=104.36.82.2:4729 , piece_id=2EW64YCU54XTYJSCKNHEZ7OTEGPNSMHPV4Q3Z3BUH5IUSV7WVOVA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:51548->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341864106 (58h8m52.186665628s) [noise=true , address=104.36.82.3:4765 , piece_id=GP2C2PTQ6TYUZE2QKEK7ZVMZJLSWAV3NLOTOEAKUVGUHCQYB57OQ, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:47384->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341909603 (58h8m52.186711143s) [seg_pos={0 0}, upload=55694]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:44:09.368838649 (58h8m52.213640171s) [upload=55694]: Done uploading segments... Jul 26 21:44:09.368863899 (58h8m52.213665413s) [upload=55694]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.041457468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................X.XX...X.X.] duration: 8.446502691s FAIL (2025-07-26 21:44:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=55680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: rpc: dial tcp 104.36.82.2:4732: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:47.771002362 (58h7m30.615803892s) [seg_pos={0 0}, upload=55680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:42:47.771184279 (58h7m30.615985833s) [upload=55680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:42:47.771192227 (58h7m30.615993753s) [upload=55680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:42:47.814292768 (58h7m30.659094298s) [upload=55680]: Done uploading segments... Jul 26 21:42:47.814316231 (58h7m30.65911776s) [upload=55680]: Upload finished. err: <nil> Jul 26 21:43:39.374613126 (58h8m22.219414639s) [upload=55694]: Starting upload Jul 26 21:43:39.374680877 (58h8m22.219482401s) [upload=55694]: Done starting upload Jul 26 21:43:39.374730849 (58h8m22.219532359s) [upload=55694]: Uploading segments... Jul 26 21:43:39.374965546 (58h8m22.219767058s) [upload=55694]: Got next segment. Inline: false Jul 26 21:43:39.375010672 (58h8m22.219812183s) [upload=55694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:43:39.375035700 (58h8m22.219837223s) [upload=55694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:43:39.400062014 (58h8m22.244863545s) [seg_pos={0 0}, upload=55694]: Begin upload segment... Jul 26 21:43:39.400128495 (58h8m22.244930025s) [seg_pos={0 0}, upload=55694]: Done begin upload segment. Jul 26 21:43:39.400297010 (58h8m22.245098546s) [noise=true , address=104.36.82.2:4729 , piece_id=2EW64YCU54XTYJSCKNHEZ7OTEGPNSMHPV4Q3Z3BUH5IUSV7WVOVA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.401011243 (58h8m22.245812775s) [noise=true , address=45.91.116.9:4261 , piece_id=NTI5BHSVNRQA3XEOSBDEFVCIIVL6JYSJMU7TBKRYGCHH3ORNMTOA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.401089603 (58h8m22.245891141s) [noise=true , address=104.36.82.6:4875 , piece_id=R2U7D4HYLWFRUYHOTYEYQATGGJAYAFB55V6XAWRVTMFR5DGXV23Q, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.402178452 (58h8m22.246979977s) [noise=true , address=45.91.116.8:4047 , piece_id=67P2ZZO3CI6WA5LUDE3GGF4F7DEEONZQ7X6FK5RXGB5FC3NPRNTA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.402248273 (58h8m22.247049795s) [noise=true , address=45.91.116.12:4227 , piece_id=EUJ6RI2H2P4AS44LKGLR7BUZ4BH3THAFZKACRRXAAI3HVWB6JSHA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403113183 (58h8m22.247914716s) [noise=true , address=77.90.31.17:4770 , piece_id=4VG4ZRU3YI7UTL4SGTWUOMHV6R7TXWNSS5LSVPDC5LOWCSOMO53A, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403289478 (58h8m22.248091001s) [noise=true , address=45.91.116.13:4155 , piece_id=OAYNJLFC2VOQTZLPGOV57SN7KPBJGCBIHH7RLN4OE5WFX345ISIA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403971148 (58h8m22.24877267s) [noise=true , address=104.36.82.12:4151 , piece_id=XAQ4DZ3B7YRQR2R5Z4TKZ65AGIKPKTJBMGZH6TDVWIVJAH5MRHEA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.404780442 (58h8m22.249581965s) [noise=true , address=45.91.116.11:4117 , piece_id=K267FD7VGIV6QOUP6A42KD3NHWY5LSOGBHK7D5BDREVHLLWH6QBA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.406562224 (58h8m22.251363756s) [noise=true , address=45.91.116.13:4153 , piece_id=FA5EMKYH7FSDYSQ4ZMUDB55G2OP4HQAFS7EATB6FHOIJIFQEQLJQ, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.408084243 (58h8m22.252885758s) [noise=true , address=77.90.31.18:4805 , piece_id=UZROHZAWFFXDODGQ4BQ3AVOBL3XVOT22D7FBSTQTUGIUAJJ4O56A, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.414472700 (58h8m22.259274213s) [noise=true , address=45.91.116.5:4297 , piece_id=UFXGLSYAJHN3WQR42P5JJ43YK5C76SX2LEHWB5DJXEK5MFNRRTTA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.417180066 (58h8m22.261981582s) [noise=true , address=104.36.82.3:4765 , piece_id=GP2C2PTQ6TYUZE2QKEK7ZVMZJLSWAV3NLOTOEAKUVGUHCQYB57OQ, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.419304498 (58h8m22.264106014s) [noise=true , address=45.91.116.14:4191 , piece_id=J2GRG7KNFB5ONW3A7MG5F6ZBC2GUARYD4PDOASHALUCIGMQJ3PMA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.421292752 (58h8m22.266094267s) [noise=true , address=45.91.116.14:4190 , piece_id=WLAJRLI2ZY5J4SFQZT5PEQTJBFZVVUVXBHG7ERGBJ5FWDAFOI2SA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.423400915 (58h8m22.268202424s) [noise=true , address=45.91.116.8:4045 , piece_id=BNWRAYUXYSSPC57U3U526OJWOXLBWUMHMLFUBNW5HVMFY4E5MRRA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.493673489 (58h8m22.338475015s) [upload=55694]: Next returned nil segment Jul 26 21:43:39.493698457 (58h8m22.33849999s) [upload=55694]: Waiting for error group managing segments... Jul 26 21:43:39.497547530 (58h8m22.342349039s) [noise=true , address=45.91.116.8:4045 , piece_id=BNWRAYUXYSSPC57U3U526OJWOXLBWUMHMLFUBNW5HVMFY4E5MRRA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.498138143 (58h8m22.342939667s) [noise=true , address=45.91.116.5:4297 , piece_id=UFXGLSYAJHN3WQR42P5JJ43YK5C76SX2LEHWB5DJXEK5MFNRRTTA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.498607426 (58h8m22.343408935s) [noise=true , address=45.91.116.13:4153 , piece_id=FA5EMKYH7FSDYSQ4ZMUDB55G2OP4HQAFS7EATB6FHOIJIFQEQLJQ, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499012857 (58h8m22.343814366s) [noise=true , address=45.91.116.9:4261 , piece_id=NTI5BHSVNRQA3XEOSBDEFVCIIVL6JYSJMU7TBKRYGCHH3ORNMTOA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499439665 (58h8m22.344241174s) [noise=true , address=45.91.116.13:4155 , piece_id=OAYNJLFC2VOQTZLPGOV57SN7KPBJGCBIHH7RLN4OE5WFX345ISIA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499977307 (58h8m22.34477883s) [noise=true , address=45.91.116.14:4190 , piece_id=WLAJRLI2ZY5J4SFQZT5PEQTJBFZVVUVXBHG7ERGBJ5FWDAFOI2SA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.500549087 (58h8m22.345350624s) [noise=true , address=45.91.116.11:4117 , piece_id=K267FD7VGIV6QOUP6A42KD3NHWY5LSOGBHK7D5BDREVHLLWH6QBA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501023407 (58h8m22.345824929s) [noise=true , address=45.91.116.8:4047 , piece_id=67P2ZZO3CI6WA5LUDE3GGF4F7DEEONZQ7X6FK5RXGB5FC3NPRNTA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501481940 (58h8m22.346283472s) [noise=true , address=45.91.116.12:4227 , piece_id=EUJ6RI2H2P4AS44LKGLR7BUZ4BH3THAFZKACRRXAAI3HVWB6JSHA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501986448 (58h8m22.346787991s) [noise=true , address=45.91.116.14:4191 , piece_id=J2GRG7KNFB5ONW3A7MG5F6ZBC2GUARYD4PDOASHALUCIGMQJ3PMA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:43.671096078 (58h8m26.515897595s) [noise=true , address=77.90.31.17:4770 , piece_id=4VG4ZRU3YI7UTL4SGTWUOMHV6R7TXWNSS5LSVPDC5LOWCSOMO53A, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:49.036287942 (58h8m31.881089454s) [noise=true , address=77.90.31.18:4805 , piece_id=UZROHZAWFFXDODGQ4BQ3AVOBL3XVOT22D7FBSTQTUGIUAJJ4O56A, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:44:09.341718713 (58h8m52.186520228s) [noise=true , address=104.36.82.12:4151 , piece_id=XAQ4DZ3B7YRQR2R5Z4TKZ65AGIKPKTJBMGZH6TDVWIVJAH5MRHEA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:47272->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341791849 (58h8m52.186593361s) [noise=true , address=104.36.82.6:4875 , piece_id=R2U7D4HYLWFRUYHOTYEYQATGGJAYAFB55V6XAWRVTMFR5DGXV23Q, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:50054->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341847201 (58h8m52.186648712s) [noise=true , address=104.36.82.2:4729 , piece_id=2EW64YCU54XTYJSCKNHEZ7OTEGPNSMHPV4Q3Z3BUH5IUSV7WVOVA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:51548->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341864106 (58h8m52.186665628s) [noise=true , address=104.36.82.3:4765 , piece_id=GP2C2PTQ6TYUZE2QKEK7ZVMZJLSWAV3NLOTOEAKUVGUHCQYB57OQ, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:47384->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341909603 (58h8m52.186711143s) [seg_pos={0 0}, upload=55694]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:44:09.368838649 (58h8m52.213640171s) [upload=55694]: Done uploading segments... Jul 26 21:44:09.368863899 (58h8m52.213665413s) [upload=55694]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.446536039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................X.XX...X.X] duration: 30.028893409s FAIL (2025-07-26 21:44:09 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=55680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: rpc: dial tcp 104.36.82.2:4732: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:47.771002362 (58h7m30.615803892s) [seg_pos={0 0}, upload=55680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:42:47.771184279 (58h7m30.615985833s) [upload=55680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:42:47.771192227 (58h7m30.615993753s) [upload=55680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:42:47.814292768 (58h7m30.659094298s) [upload=55680]: Done uploading segments... Jul 26 21:42:47.814316231 (58h7m30.65911776s) [upload=55680]: Upload finished. err: <nil> Jul 26 21:43:39.374613126 (58h8m22.219414639s) [upload=55694]: Starting upload Jul 26 21:43:39.374680877 (58h8m22.219482401s) [upload=55694]: Done starting upload Jul 26 21:43:39.374730849 (58h8m22.219532359s) [upload=55694]: Uploading segments... Jul 26 21:43:39.374965546 (58h8m22.219767058s) [upload=55694]: Got next segment. Inline: false Jul 26 21:43:39.375010672 (58h8m22.219812183s) [upload=55694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:43:39.375035700 (58h8m22.219837223s) [upload=55694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:43:39.400062014 (58h8m22.244863545s) [seg_pos={0 0}, upload=55694]: Begin upload segment... Jul 26 21:43:39.400128495 (58h8m22.244930025s) [seg_pos={0 0}, upload=55694]: Done begin upload segment. Jul 26 21:43:39.400297010 (58h8m22.245098546s) [noise=true , address=104.36.82.2:4729 , piece_id=2EW64YCU54XTYJSCKNHEZ7OTEGPNSMHPV4Q3Z3BUH5IUSV7WVOVA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.401011243 (58h8m22.245812775s) [noise=true , address=45.91.116.9:4261 , piece_id=NTI5BHSVNRQA3XEOSBDEFVCIIVL6JYSJMU7TBKRYGCHH3ORNMTOA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.401089603 (58h8m22.245891141s) [noise=true , address=104.36.82.6:4875 , piece_id=R2U7D4HYLWFRUYHOTYEYQATGGJAYAFB55V6XAWRVTMFR5DGXV23Q, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.402178452 (58h8m22.246979977s) [noise=true , address=45.91.116.8:4047 , piece_id=67P2ZZO3CI6WA5LUDE3GGF4F7DEEONZQ7X6FK5RXGB5FC3NPRNTA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.402248273 (58h8m22.247049795s) [noise=true , address=45.91.116.12:4227 , piece_id=EUJ6RI2H2P4AS44LKGLR7BUZ4BH3THAFZKACRRXAAI3HVWB6JSHA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403113183 (58h8m22.247914716s) [noise=true , address=77.90.31.17:4770 , piece_id=4VG4ZRU3YI7UTL4SGTWUOMHV6R7TXWNSS5LSVPDC5LOWCSOMO53A, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403289478 (58h8m22.248091001s) [noise=true , address=45.91.116.13:4155 , piece_id=OAYNJLFC2VOQTZLPGOV57SN7KPBJGCBIHH7RLN4OE5WFX345ISIA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.403971148 (58h8m22.24877267s) [noise=true , address=104.36.82.12:4151 , piece_id=XAQ4DZ3B7YRQR2R5Z4TKZ65AGIKPKTJBMGZH6TDVWIVJAH5MRHEA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.404780442 (58h8m22.249581965s) [noise=true , address=45.91.116.11:4117 , piece_id=K267FD7VGIV6QOUP6A42KD3NHWY5LSOGBHK7D5BDREVHLLWH6QBA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.406562224 (58h8m22.251363756s) [noise=true , address=45.91.116.13:4153 , piece_id=FA5EMKYH7FSDYSQ4ZMUDB55G2OP4HQAFS7EATB6FHOIJIFQEQLJQ, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.408084243 (58h8m22.252885758s) [noise=true , address=77.90.31.18:4805 , piece_id=UZROHZAWFFXDODGQ4BQ3AVOBL3XVOT22D7FBSTQTUGIUAJJ4O56A, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.414472700 (58h8m22.259274213s) [noise=true , address=45.91.116.5:4297 , piece_id=UFXGLSYAJHN3WQR42P5JJ43YK5C76SX2LEHWB5DJXEK5MFNRRTTA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.417180066 (58h8m22.261981582s) [noise=true , address=104.36.82.3:4765 , piece_id=GP2C2PTQ6TYUZE2QKEK7ZVMZJLSWAV3NLOTOEAKUVGUHCQYB57OQ, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.419304498 (58h8m22.264106014s) [noise=true , address=45.91.116.14:4191 , piece_id=J2GRG7KNFB5ONW3A7MG5F6ZBC2GUARYD4PDOASHALUCIGMQJ3PMA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.421292752 (58h8m22.266094267s) [noise=true , address=45.91.116.14:4190 , piece_id=WLAJRLI2ZY5J4SFQZT5PEQTJBFZVVUVXBHG7ERGBJ5FWDAFOI2SA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.423400915 (58h8m22.268202424s) [noise=true , address=45.91.116.8:4045 , piece_id=BNWRAYUXYSSPC57U3U526OJWOXLBWUMHMLFUBNW5HVMFY4E5MRRA, seg_pos={0 0}, upload=55694]: Uploading piece... Jul 26 21:43:39.493673489 (58h8m22.338475015s) [upload=55694]: Next returned nil segment Jul 26 21:43:39.493698457 (58h8m22.33849999s) [upload=55694]: Waiting for error group managing segments... Jul 26 21:43:39.497547530 (58h8m22.342349039s) [noise=true , address=45.91.116.8:4045 , piece_id=BNWRAYUXYSSPC57U3U526OJWOXLBWUMHMLFUBNW5HVMFY4E5MRRA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.498138143 (58h8m22.342939667s) [noise=true , address=45.91.116.5:4297 , piece_id=UFXGLSYAJHN3WQR42P5JJ43YK5C76SX2LEHWB5DJXEK5MFNRRTTA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.498607426 (58h8m22.343408935s) [noise=true , address=45.91.116.13:4153 , piece_id=FA5EMKYH7FSDYSQ4ZMUDB55G2OP4HQAFS7EATB6FHOIJIFQEQLJQ, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499012857 (58h8m22.343814366s) [noise=true , address=45.91.116.9:4261 , piece_id=NTI5BHSVNRQA3XEOSBDEFVCIIVL6JYSJMU7TBKRYGCHH3ORNMTOA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499439665 (58h8m22.344241174s) [noise=true , address=45.91.116.13:4155 , piece_id=OAYNJLFC2VOQTZLPGOV57SN7KPBJGCBIHH7RLN4OE5WFX345ISIA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.499977307 (58h8m22.34477883s) [noise=true , address=45.91.116.14:4190 , piece_id=WLAJRLI2ZY5J4SFQZT5PEQTJBFZVVUVXBHG7ERGBJ5FWDAFOI2SA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.500549087 (58h8m22.345350624s) [noise=true , address=45.91.116.11:4117 , piece_id=K267FD7VGIV6QOUP6A42KD3NHWY5LSOGBHK7D5BDREVHLLWH6QBA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501023407 (58h8m22.345824929s) [noise=true , address=45.91.116.8:4047 , piece_id=67P2ZZO3CI6WA5LUDE3GGF4F7DEEONZQ7X6FK5RXGB5FC3NPRNTA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501481940 (58h8m22.346283472s) [noise=true , address=45.91.116.12:4227 , piece_id=EUJ6RI2H2P4AS44LKGLR7BUZ4BH3THAFZKACRRXAAI3HVWB6JSHA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:39.501986448 (58h8m22.346787991s) [noise=true , address=45.91.116.14:4191 , piece_id=J2GRG7KNFB5ONW3A7MG5F6ZBC2GUARYD4PDOASHALUCIGMQJ3PMA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:43.671096078 (58h8m26.515897595s) [noise=true , address=77.90.31.17:4770 , piece_id=4VG4ZRU3YI7UTL4SGTWUOMHV6R7TXWNSS5LSVPDC5LOWCSOMO53A, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:43:49.036287942 (58h8m31.881089454s) [noise=true , address=77.90.31.18:4805 , piece_id=UZROHZAWFFXDODGQ4BQ3AVOBL3XVOT22D7FBSTQTUGIUAJJ4O56A, seg_pos={0 0}, upload=55694]: Done uploading piece. err: <nil> Jul 26 21:44:09.341718713 (58h8m52.186520228s) [noise=true , address=104.36.82.12:4151 , piece_id=XAQ4DZ3B7YRQR2R5Z4TKZ65AGIKPKTJBMGZH6TDVWIVJAH5MRHEA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:47272->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341791849 (58h8m52.186593361s) [noise=true , address=104.36.82.6:4875 , piece_id=R2U7D4HYLWFRUYHOTYEYQATGGJAYAFB55V6XAWRVTMFR5DGXV23Q, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:50054->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341847201 (58h8m52.186648712s) [noise=true , address=104.36.82.2:4729 , piece_id=2EW64YCU54XTYJSCKNHEZ7OTEGPNSMHPV4Q3Z3BUH5IUSV7WVOVA, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:51548->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341864106 (58h8m52.186665628s) [noise=true , address=104.36.82.3:4765 , piece_id=GP2C2PTQ6TYUZE2QKEK7ZVMZJLSWAV3NLOTOEAKUVGUHCQYB57OQ, seg_pos={0 0}, upload=55694]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:47384->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:44:09.341909603 (58h8m52.186711143s) [seg_pos={0 0}, upload=55694]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:44:09.368838649 (58h8m52.213640171s) [upload=55694]: Done uploading segments... Jul 26 21:44:09.368863899 (58h8m52.213665413s) [upload=55694]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.028915635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.......X.......X.] duration: 1.856671524s FAIL (2025-07-26 21:43:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.856696285s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 1m0.039180682s FAIL (2025-07-26 21:43:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== , piece_id=KBNW25EIZJC4GH6PGMRDPLFISBGHFWUZMQWWF4MAAX2EZOAWMHXA, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.134504405 (58h7m30.979305938s) [noise=true , address=104.36.82.56:4256 , piece_id=LBRE4QOMTZEF4NHPY3U7WGZY6NP2RLDFAMKET2MLHDKKNQOSJ62A, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.134967450 (58h7m30.979768984s) [noise=true , address=104.36.82.16:4304 , piece_id=CX4GQQXAVLZNTTVADEC7RN4H3AV2BEOGBCEEIIQW6LJCROLQ5RTA, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.135157789 (58h7m30.979959321s) [noise=true , address=104.36.82.55:4207 , piece_id=DU2QACCOTX24OKB6YWBB4SIWPUE7HRYEUUNFAEKESNAIQFEFGFWA, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.135506206 (58h7m30.980307738s) [noise=true , address=79.127.201.215:4345 , piece_id=3RGLLG4VMUM577OFRDJHBX6CQ7YWXDS7HKY3KLDSSJDYDUR5X4XQ, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.135615858 (58h7m30.980417389s) [noise=true , address=77.90.31.18:4832 , piece_id=HZS3RWZAHQODMW5M3ZIN6OSDZVLHUXSP7VNJ7HTCLL2KWDUWX3KQ, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.135976531 (58h7m30.980778065s) [noise=true , address=79.127.205.234:4611 , piece_id=BFZYNLQLBPFVLYMGJNNYRT3FWJPVB2WG7WK5Y4O4I73AV7WIVYKQ, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.136014309 (58h7m30.980816176s) [noise=true , address=104.36.82.51:4059 , piece_id=Y4BBMMTBRUXAQOOU2QIKXHRPKK6MGYUUJF2Q65DWBC3FGZIGN2SA, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.136471423 (58h7m30.981272958s) [noise=true , address=104.36.82.46:4492 , piece_id=USAUAAYANUD3ZUWHKYLFDHFONOX6LCJGBGMFM77O6VKV5EKZTIFQ, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.136542321 (58h7m30.981343853s) [noise=true , address=104.36.82.47:4543 , piece_id=FNFQPW6IKNEWQTU3DCMUY7UAIH7U3HVJK72TNYQ5WTWWEDCAQC3A, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.136964333 (58h7m30.981765864s) [noise=true , address=79.127.205.245:4243 , piece_id=AG5JJREXML3Q3T62G52XW47QYLL3DL2Q663PP6HONXSFMD4ASGFA, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.137374623 (58h7m30.98217616s) [noise=true , address=104.36.82.5:4862 , piece_id=WD4NTFTB53I7DCXUNUFYCAUPVS5LURZUYWHGNWREK6HUCCLK3S2Q, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.137738740 (58h7m30.982540273s) [noise=true , address=104.36.82.19:4411 , piece_id=XCWBGNGCXKMVU3XLWAA2FEHD527XVSCP5EGVBTO5QTPXARV77AXA, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.138058981 (58h7m30.982860508s) [noise=true , address=79.127.219.38:4015 , piece_id=BEHTB6TIOCH7VTQSO7YCHYJHVRW4IM6H7HTZQCIMABUJNENKI6XQ, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.138425817 (58h7m30.983227348s) [noise=true , address=156.146.43.225:4434 , piece_id=MP73NVUPZGINJCOSJ2PIE2Y24YK5VI35ZLOVVKQ2K5C2GHJWEGXQ, seg_pos={0 0}, upload=55681]: Uploading piece... Jul 26 21:42:48.194989609 (58h7m31.039791118s) [upload=55681]: Next returned nil segment Jul 26 21:42:48.195011003 (58h7m31.039812535s) [upload=55681]: Waiting for error group managing segments... Jul 26 21:42:48.268339330 (58h7m31.113140886s) [noise=true , address=79.127.219.38:4015 , piece_id=BEHTB6TIOCH7VTQSO7YCHYJHVRW4IM6H7HTZQCIMABUJNENKI6XQ, seg_pos={0 0}, upload=55681]: Done uploading piece. err: <nil> Jul 26 21:42:48.384106559 (58h7m31.228908081s) [noise=true , address=156.146.43.225:4434 , piece_id=MP73NVUPZGINJCOSJ2PIE2Y24YK5VI35ZLOVVKQ2K5C2GHJWEGXQ, seg_pos={0 0}, upload=55681]: Done uploading piece. err: <nil> Jul 26 21:42:48.707005312 (58h7m31.551806831s) [noise=true , address=79.127.205.245:4243 , piece_id=AG5JJREXML3Q3T62G52XW47QYLL3DL2Q663PP6HONXSFMD4ASGFA, seg_pos={0 0}, upload=55681]: Done uploading piece. err: <nil> Jul 26 21:42:48.724728966 (58h7m31.569530488s) [noise=true , address=79.127.205.234:4611 , piece_id=BFZYNLQLBPFVLYMGJNNYRT3FWJPVB2WG7WK5Y4O4I73AV7WIVYKQ, seg_pos={0 0}, upload=55681]: Done uploading piece. err: <nil> Jul 26 21:42:48.805273771 (58h7m31.650075294s) [noise=true , address=64.49.13.170:4719 , piece_id=NHSHHPGRVJKQYQV26IYQOQSSG3VJBWLITRWNLDR7A7KS54BB6D2Q, seg_pos={0 0}, upload=55681]: Done uploading piece. err: <nil> Jul 26 21:42:52.570233097 (58h7m35.415034608s) [noise=true , address=77.90.31.18:4832 , piece_id=HZS3RWZAHQODMW5M3ZIN6OSDZVLHUXSP7VNJ7HTCLL2KWDUWX3KQ, seg_pos={0 0}, upload=55681]: Done uploading piece. err: <nil> Jul 26 21:42:55.127973982 (58h7m37.972775498s) [noise=true , address=79.127.201.215:4345 , piece_id=3RGLLG4VMUM577OFRDJHBX6CQ7YWXDS7HKY3KLDSSJDYDUR5X4XQ, seg_pos={0 0}, upload=55681]: Done uploading piece. err: <nil> Jul 26 21:43:48.057269423 (58h8m30.902070934s) [noise=true , address=104.36.82.16:4304 , piece_id=CX4GQQXAVLZNTTVADEC7RN4H3AV2BEOGBCEEIIQW6LJCROLQ5RTA, seg_pos={0 0}, upload=55681]: Done uploading piece. err: ecclient: upload failed (node:12FNMhT2mqrLmJKfLtqnYqE7hpz4AzaCskjUp6oVg3dDgdau2Rv, address:104.36.82.16:4304): protocol: rpc: dial tcp 104.36.82.16:4304: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:43:48.057348519 (58h8m30.902150032s) [noise=true , address=104.36.82.55:4207 , piece_id=DU2QACCOTX24OKB6YWBB4SIWPUE7HRYEUUNFAEKESNAIQFEFGFWA, seg_pos={0 0}, upload=55681]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: rpc: dial tcp 104.36.82.55:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:43:48.057403858 (58h8m30.90220537s) [noise=true , address=104.36.82.47:4543 , piece_id=FNFQPW6IKNEWQTU3DCMUY7UAIH7U3HVJK72TNYQ5WTWWEDCAQC3A, seg_pos={0 0}, upload=55681]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: rpc: dial tcp 104.36.82.47:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:43:48.057456129 (58h8m30.902257641s) [noise=true , address=104.36.82.19:4411 , piece_id=XCWBGNGCXKMVU3XLWAA2FEHD527XVSCP5EGVBTO5QTPXARV77AXA, seg_pos={0 0}, upload=55681]: Done uploading piece. err: ecclient: upload failed (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd, address:104.36.82.19:4411): protocol: rpc: dial tcp 104.36.82.19:4411: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:43:48.057503501 (58h8m30.90230502s) [noise=true , address=104.36.82.38:4723 , piece_id=G5L627AREMNEZLISZVSEZMV7BJSXEDB7ZBZSHRN3YFAI7SCWITWQ, seg_pos={0 0}, upload=55681]: Done uploading piece. err: ecclient: upload failed (node:1SBKU1E8RjvAx3NzWEveNX5cfuQqq4SKgzY15eeJyfMV2DcF9i, address:104.36.82.38:4723): protocol: rpc: dial tcp 104.36.82.38:4723: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:43:48.057561860 (58h8m30.902363372s) [noise=true , address=104.36.82.56:4256 , piece_id=LBRE4QOMTZEF4NHPY3U7WGZY6NP2RLDFAMKET2MLHDKKNQOSJ62A, seg_pos={0 0}, upload=55681]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: rpc: dial tcp 104.36.82.56:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:43:48.057616493 (58h8m30.902418016s) [noise=true , address=104.36.82.46:4492 , piece_id=USAUAAYANUD3ZUWHKYLFDHFONOX6LCJGBGMFM77O6VKV5EKZTIFQ, seg_pos={0 0}, upload=55681]: Done uploading piece. err: ecclient: upload failed (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip, address:104.36.82.46:4492): protocol: rpc: dial tcp 104.36.82.46:4492: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:43:48.057634188 (58h8m30.902435715s) [noise=true , address=104.36.82.49:4984 , piece_id=UNTZR7WNYJBETPT65NAMDKERGSHS7S3FCCW2NX4GUZEZA6RQNQOA, seg_pos={0 0}, upload=55681]: Done uploading piece. err: ecclient: upload failed (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv, address:104.36.82.49:4984): protocol: write tcp 159.223.143.39:47894->104.36.82.49:4984: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:43:48.057707262 (58h8m30.902508787s) [noise=true , address=104.36.82.5:4862 , piece_id=WD4NTFTB53I7DCXUNUFYCAUPVS5LURZUYWHGNWREK6HUCCLK3S2Q, seg_pos={0 0}, upload=55681]: Done uploading piece. err: ecclient: upload failed (node:12sUT7LBYFqWMUhx7MMjEmyiFDnbJnjSVndNbTaHioaVLLSXYue, address:104.36.82.5:4862): protocol: rpc: dial tcp 104.36.82.5:4862: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:43:48.057714029 (58h8m30.902515554s) [noise=true , address=104.36.82.51:4059 , piece_id=Y4BBMMTBRUXAQOOU2QIKXHRPKK6MGYUUJF2Q65DWBC3FGZIGN2SA, seg_pos={0 0}, upload=55681]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:43:48.057763595 (58h8m30.902565109s) [noise=true , address=104.36.82.2:4756 , piece_id=KBNW25EIZJC4GH6PGMRDPLFISBGHFWUZMQWWF4MAAX2EZOAWMHXA, seg_pos={0 0}, upload=55681]: Done uploading piece. err: ecclient: upload failed (node:1Pow7VVb1PRDBvHjEcS4bfnh3276S313jiGawszkhaJ582pDqQ, address:104.36.82.2:4756): protocol: write tcp 159.223.143.39:49880->104.36.82.2:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:43:48.057794816 (58h8m30.90259633s) [seg_pos={0 0}, upload=55681]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 21:43:48.093952526 (58h8m30.938754053s) [upload=55681]: Done uploading segments... Jul 26 21:43:48.094017154 (58h8m30.938818681s) [upload=55681]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.039276446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.......X.......X] duration: 30.050681893s FAIL (2025-07-26 21:43:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.050707701s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.......X.......] duration: 397.862329ms FAIL (2025-07-26 21:42:51 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 397.897268ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................X.XX...X.] duration: 5.32727323s FAIL (2025-07-26 21:42:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== :4221 , piece_id=TRRA2ZWGGFM6445C4NMSWYO5H73KTY3OPQH3CM7U55BX74UHQI7Q, seg_pos={0 0}, upload=55658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 159.223.143.39:55862->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Jul 26 21:41:31.132553832 (58h6m13.977355344s) [seg_pos={0 0}, upload=55658]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:41:31.132636072 (58h6m13.977437594s) [upload=55658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:41:31.132642523 (58h6m13.977444046s) [upload=55658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:41:31.189100166 (58h6m14.033901688s) [upload=55658]: Done uploading segments... Jul 26 21:41:31.189118910 (58h6m14.033920432s) [upload=55658]: Upload finished. err: <nil> Jul 26 21:41:43.593978116 (58h6m26.438779633s) [upload=55669]: Starting upload Jul 26 21:41:43.594091278 (58h6m26.438892865s) [upload=55669]: Done starting upload Jul 26 21:41:43.594198711 (58h6m26.439000223s) [upload=55669]: Uploading segments... Jul 26 21:41:43.594617474 (58h6m26.439418997s) [upload=55669]: Got next segment. Inline: false Jul 26 21:41:43.594688129 (58h6m26.439489654s) [upload=55669]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:41:43.594695983 (58h6m26.439497507s) [upload=55669]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:41:43.676662603 (58h6m26.521464134s) [seg_pos={0 0}, upload=55669]: Begin upload segment... Jul 26 21:41:43.676732330 (58h6m26.521533843s) [seg_pos={0 0}, upload=55669]: Done begin upload segment. Jul 26 21:41:43.676761988 (58h6m26.521563521s) [noise=true , address=77.90.31.16:4735 , piece_id=6A3ATQE7FGUR2W6VV72MKVDHFB224PAFUWJHULNQUARZVV47A7CA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.676811055 (58h6m26.521612631s) [noise=true , address=45.91.116.8:4045 , piece_id=V5ZTHGV5XQDLJVA46FGLZKVWRAQHWZANIHML2MM5C5IJXXW6YBGA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.677223825 (58h6m26.522025347s) [noise=true , address=45.91.116.12:4227 , piece_id=QOI37VENLHL74VCVTLSLUXCDQJ5ENHXYREVAYK7NBAMRKLBOTVZQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.677352278 (58h6m26.522153806s) [noise=true , address=45.91.116.8:4047 , piece_id=ONEQP4LIGA33BD6TJPUOKU3RYZUPRMHAYRA6UJFUNA27IODCMEBQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.677584306 (58h6m26.522385831s) [noise=true , address=104.36.82.7:4911 , piece_id=ZL7GXTUTDEHKOAV443PUZF6AZGO2RWVUYKIKDSF5IUQMKENC5YKA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.677836843 (58h6m26.522638374s) [noise=true , address=77.90.31.15:4699 , piece_id=JAKHHUQTJS6PGMPRK4TM34Z6AF4RJ47Q2SXSK2EB724K52M2FPZA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.677907949 (58h6m26.522709481s) [noise=true , address=45.91.116.10:4083 , piece_id=2FLQNVCAOLQJJEFKLKBF7LT76SYYM7NYKJ4CQX6TCDJISNFTJZJA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.678222879 (58h6m26.523024404s) [noise=true , address=104.36.82.15:4259 , piece_id=ATXEGVAJORFE2RZXLEJHCOEUO6QW2AC6LUFLOQ44UVA32JOSEL6A, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.678338388 (58h6m26.52313992s) [noise=true , address=45.91.116.9:4263 , piece_id=TO6JLKYMOWLSFWTQZ5XDUT3MX7EPVMUUWN6XIWRHCJWQYKHTFA2Q, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.678576793 (58h6m26.523378315s) [noise=true , address=104.36.82.13:4186 , piece_id=GEVIXCPVJQ36MZXZ2MFIZ4DTXJVDYNBU5T7JPEX7M2XLIVVHHEQA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.678810262 (58h6m26.52361179s) [noise=true , address=45.91.116.6:4974 , piece_id=6K32JH3TE2AATAPT3B4ZV2W6JLYTV6XZ5BEOZJMPLQZVC5LVNRFA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.678902110 (58h6m26.523703632s) [noise=true , address=45.91.116.14:4190 , piece_id=SNIE3P5ELZZEGRMAWSSG7JW45MRCPJGJIBJP5PLRLPJ4XQ4NVDMQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.679251743 (58h6m26.524053274s) [noise=true , address=77.90.31.19:4842 , piece_id=DR4YEDADRAGGGOB6WBTOW32TY3YMDNVWGC3LVS2HTK3PZ7TYIJXQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.679258304 (58h6m26.524059838s) [noise=true , address=77.90.31.13:4628 , piece_id=RYQ6QPT7CNGA4E74QLISOFLRW6X7YPZUNYUCQRK2UB6OTNFO3FSQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.679716730 (58h6m26.524518264s) [noise=true , address=104.36.82.2:4732 , piece_id=LQCEG6ITTPOS74NFBST4XYUKEAXBX43KGGMLWN4INU3ISZFDN4GA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.680151303 (58h6m26.524952828s) [noise=true , address=45.91.116.5:4297 , piece_id=GOASWSTM7SNXNV6WTCNXTE6ZFULVZVNTDKB7YNEM7FP6TLWD2SXQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.756082686 (58h6m26.600884201s) [upload=55669]: Next returned nil segment Jul 26 21:41:43.756098979 (58h6m26.600900492s) [upload=55669]: Waiting for error group managing segments... Jul 26 21:41:43.760489386 (58h6m26.605290901s) [noise=true , address=45.91.116.9:4263 , piece_id=TO6JLKYMOWLSFWTQZ5XDUT3MX7EPVMUUWN6XIWRHCJWQYKHTFA2Q, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.761122019 (58h6m26.605923558s) [noise=true , address=45.91.116.5:4297 , piece_id=GOASWSTM7SNXNV6WTCNXTE6ZFULVZVNTDKB7YNEM7FP6TLWD2SXQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.761589275 (58h6m26.606390791s) [noise=true , address=45.91.116.6:4974 , piece_id=6K32JH3TE2AATAPT3B4ZV2W6JLYTV6XZ5BEOZJMPLQZVC5LVNRFA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.761986968 (58h6m26.606788479s) [noise=true , address=45.91.116.8:4047 , piece_id=ONEQP4LIGA33BD6TJPUOKU3RYZUPRMHAYRA6UJFUNA27IODCMEBQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.762412003 (58h6m26.607213529s) [noise=true , address=45.91.116.14:4190 , piece_id=SNIE3P5ELZZEGRMAWSSG7JW45MRCPJGJIBJP5PLRLPJ4XQ4NVDMQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.762853910 (58h6m26.607655421s) [noise=true , address=45.91.116.12:4227 , piece_id=QOI37VENLHL74VCVTLSLUXCDQJ5ENHXYREVAYK7NBAMRKLBOTVZQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.763284522 (58h6m26.608086048s) [noise=true , address=45.91.116.8:4045 , piece_id=V5ZTHGV5XQDLJVA46FGLZKVWRAQHWZANIHML2MM5C5IJXXW6YBGA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.763664187 (58h6m26.608465698s) [noise=true , address=45.91.116.10:4083 , piece_id=2FLQNVCAOLQJJEFKLKBF7LT76SYYM7NYKJ4CQX6TCDJISNFTJZJA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:45.896045410 (58h6m28.740846949s) [noise=true , address=77.90.31.15:4699 , piece_id=JAKHHUQTJS6PGMPRK4TM34Z6AF4RJ47Q2SXSK2EB724K52M2FPZA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:46.907240573 (58h6m29.752042099s) [noise=true , address=77.90.31.16:4735 , piece_id=6A3ATQE7FGUR2W6VV72MKVDHFB224PAFUWJHULNQUARZVV47A7CA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:47.849297242 (58h6m30.694098769s) [noise=true , address=77.90.31.19:4842 , piece_id=DR4YEDADRAGGGOB6WBTOW32TY3YMDNVWGC3LVS2HTK3PZ7TYIJXQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:54.264479096 (58h6m37.10928061s) [noise=true , address=77.90.31.13:4628 , piece_id=RYQ6QPT7CNGA4E74QLISOFLRW6X7YPZUNYUCQRK2UB6OTNFO3FSQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:42:13.556173009 (58h6m56.400974538s) [noise=true , address=104.36.82.2:4732 , piece_id=LQCEG6ITTPOS74NFBST4XYUKEAXBX43KGGMLWN4INU3ISZFDN4GA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:13.556429672 (58h6m56.401231199s) [noise=true , address=104.36.82.15:4259 , piece_id=ATXEGVAJORFE2RZXLEJHCOEUO6QW2AC6LUFLOQ44UVA32JOSEL6A, seg_pos={0 0}, upload=55669]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:13.556505874 (58h6m56.401307399s) [noise=true , address=104.36.82.7:4911 , piece_id=ZL7GXTUTDEHKOAV443PUZF6AZGO2RWVUYKIKDSF5IUQMKENC5YKA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:13.556597462 (58h6m56.40139899s) [noise=true , address=104.36.82.13:4186 , piece_id=GEVIXCPVJQ36MZXZ2MFIZ4DTXJVDYNBU5T7JPEX7M2XLIVVHHEQA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:36276->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:13.556673050 (58h6m56.401474584s) [seg_pos={0 0}, upload=55669]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:42:13.581818490 (58h6m56.426620015s) [upload=55669]: Done uploading segments... Jul 26 21:42:13.581886792 (58h6m56.426688319s) [upload=55669]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.327295391s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 1m0.038624539s FAIL (2025-07-26 21:42:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== EDKXWWHIVHX4ZDNMVKYNY6MZ3TQF5IV6V6RAGRKRSYDJ4JQ, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.593239968 (58h6m20.4380415s) [noise=true , address=104.36.82.54:4174 , piece_id=3MMDELCHGHTAPZFVDUMTXYVW3UZKUGW52JVGWJCFBNLRU6HSVRFA, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.593291529 (58h6m20.43809306s) [noise=true , address=104.36.82.41:4775 , piece_id=63M37DWDU3I3IFIXFNK2HSCHZOQFVDSO63WAO3PBYRDNZLHABYLQ, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.593571531 (58h6m20.438373065s) [noise=true , address=104.36.82.12:4153 , piece_id=DRNCGMSUW3ZFJAQ6LBMPHOOTFROVBTZP3JK3GQDY7VP3RG3JKICQ, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.593772581 (58h6m20.438574118s) [noise=true , address=79.127.205.246:4282 , piece_id=UFF23QVP4CH4IIGLRP44LWEVYMDFQ4MLV6IBICPFRVJGJS6DAOXQ, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.593920729 (58h6m20.438722237s) [noise=true , address=104.36.82.53:4129 , piece_id=NN6PCFKR2H56TC45GJ7RM7KMFJ2BGVOPZ2S5LNLG7BTQDOC6KPIA, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.594270200 (58h6m20.439071735s) [noise=true , address=104.36.82.46:4508 , piece_id=XTTEK5344LKPE5A266WVBKXQSF4KGRNG4CCSXWLVZBD73USQITVQ, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.594770178 (58h6m20.439571711s) [noise=true , address=104.36.82.22:4518 , piece_id=2MTUBCWNFDL6TP4OKWQCKEMOHUCP2RB2XMSAZUUDXW5PJVOZSHKA, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.595390830 (58h6m20.440192362s) [noise=true , address=104.36.82.52:4089 , piece_id=WFSYKI35PFJ75W6XTTX2ATRNFNEJY6H4BH72G66UIPFVERVLWKQQ, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.595883568 (58h6m20.440685108s) [noise=true , address=104.36.82.7:4926 , piece_id=TFEUKZFIXSLRCWMDHLI5D3YNRVKAGBSMRUDJOL4TQSAP4YMGQMZA, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.597525028 (58h6m20.442326557s) [noise=true , address=79.127.201.215:4343 , piece_id=P67RF3M67D3PWYYS7QIS7UK6WZDWPZT3BZW7NCIQNLPOKSUW2XDA, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.597935068 (58h6m20.44273662s) [noise=true , address=104.36.82.55:4200 , piece_id=EC6BX4YWAQWSUULTBJBVAEEVPYRO3WUEUEFR7TXE6BXVRGBO6XSA, seg_pos={0 0}, upload=55667]: Uploading piece... Jul 26 21:41:37.810921745 (58h6m20.655723275s) [upload=55667]: Next returned nil segment Jul 26 21:41:37.810946463 (58h6m20.655747998s) [upload=55667]: Waiting for error group managing segments... Jul 26 21:41:38.217718764 (58h6m21.062520286s) [noise=true , address=79.127.205.246:4282 , piece_id=UFF23QVP4CH4IIGLRP44LWEVYMDFQ4MLV6IBICPFRVJGJS6DAOXQ, seg_pos={0 0}, upload=55667]: Done uploading piece. err: <nil> Jul 26 21:41:40.122911849 (58h6m22.967713365s) [noise=true , address=79.127.201.215:4343 , piece_id=P67RF3M67D3PWYYS7QIS7UK6WZDWPZT3BZW7NCIQNLPOKSUW2XDA, seg_pos={0 0}, upload=55667]: Done uploading piece. err: <nil> Jul 26 21:41:43.605326551 (58h6m26.450128082s) [noise=true , address=77.90.31.21:4912 , piece_id=VOMT3W6EBTSGGESCUWCPPOL7WXB5CJKW7HRBI6DLESX27OZWQXRQ, seg_pos={0 0}, upload=55667]: Done uploading piece. err: <nil> Jul 26 21:41:45.942622842 (58h6m28.787424395s) [noise=true , address=77.90.31.10:4530 , piece_id=YICXDOJ4MWW5HKQSIILV5GM7O6YO5QPPE3QCI5RP7KLL75VKIR7Q, seg_pos={0 0}, upload=55667]: Done uploading piece. err: <nil> Jul 26 21:42:37.527527560 (58h7m20.372329092s) [noise=true , address=104.36.82.41:4775 , piece_id=63M37DWDU3I3IFIXFNK2HSCHZOQFVDSO63WAO3PBYRDNZLHABYLQ, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj, address:104.36.82.41:4775): protocol: rpc: dial tcp 104.36.82.41:4775: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.527619220 (58h7m20.372420742s) [noise=true , address=104.36.82.22:4518 , piece_id=2MTUBCWNFDL6TP4OKWQCKEMOHUCP2RB2XMSAZUUDXW5PJVOZSHKA, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:1ozLDHDa9ankBimUBjpM91VrraKMptSCQPoDS9yWu2tjVErasU, address:104.36.82.22:4518): protocol: rpc: dial tcp 104.36.82.22:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.527672736 (58h7m20.372474258s) [noise=true , address=104.36.82.52:4089 , piece_id=WFSYKI35PFJ75W6XTTX2ATRNFNEJY6H4BH72G66UIPFVERVLWKQQ, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: rpc: dial tcp 104.36.82.52:4089: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.527721497 (58h7m20.37252302s) [noise=true , address=104.36.82.11:4136 , piece_id=NGP7YVNB7IH367RZA7VAKDZ3A5OMYU4PH3WEIGOAWQOPF3LJJTEQ, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:127n8CMbZQ7haiv6tGHrVg9V8Yf12sYkcbL9mEbSpQ4WkT9isnp, address:104.36.82.11:4136): protocol: rpc: dial tcp 104.36.82.11:4136: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.527771053 (58h7m20.372572576s) [noise=true , address=104.36.82.46:4483 , piece_id=TZOW4H5OGGACDBXWQLGE26X2Z2WWKRRHISKOAEHAB3ICUCVEQBUA, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, address:104.36.82.46:4483): protocol: rpc: dial tcp 104.36.82.46:4483: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.527820374 (58h7m20.3726219s) [noise=true , address=104.36.82.54:4174 , piece_id=3MMDELCHGHTAPZFVDUMTXYVW3UZKUGW52JVGWJCFBNLRU6HSVRFA, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: rpc: dial tcp 104.36.82.54:4174: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.527867915 (58h7m20.372669438s) [noise=true , address=104.36.82.53:4129 , piece_id=NN6PCFKR2H56TC45GJ7RM7KMFJ2BGVOPZ2S5LNLG7BTQDOC6KPIA, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: rpc: dial tcp 104.36.82.53:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.527924896 (58h7m20.372726419s) [noise=true , address=104.36.82.7:4926 , piece_id=TFEUKZFIXSLRCWMDHLI5D3YNRVKAGBSMRUDJOL4TQSAP4YMGQMZA, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: rpc: dial tcp 104.36.82.7:4926: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.527970078 (58h7m20.3727716s) [noise=true , address=104.36.82.55:4200 , piece_id=EC6BX4YWAQWSUULTBJBVAEEVPYRO3WUEUEFR7TXE6BXVRGBO6XSA, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: rpc: dial tcp 104.36.82.55:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.528015695 (58h7m20.372817218s) [noise=true , address=104.36.82.12:4153 , piece_id=DRNCGMSUW3ZFJAQ6LBMPHOOTFROVBTZP3JK3GQDY7VP3RG3JKICQ, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:124myE1wFhvjA5paJy5tr2hRe8jZCoEdbJLdBwvss1ZnT4GvtZv, address:104.36.82.12:4153): protocol: rpc: dial tcp 104.36.82.12:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.528062729 (58h7m20.372864252s) [noise=true , address=104.36.82.46:4508 , piece_id=XTTEK5344LKPE5A266WVBKXQSF4KGRNG4CCSXWLVZBD73USQITVQ, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: rpc: dial tcp 104.36.82.46:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.528107324 (58h7m20.372908847s) [noise=true , address=104.36.82.20:4444 , piece_id=MPKUJT3HSYX3THMP7PU4TPJNTGXYZVKEDRMM32SNWSMEM4G4X4BQ, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:4444): protocol: rpc: dial tcp 104.36.82.20:4444: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.528152236 (58h7m20.372953759s) [noise=true , address=104.36.82.47:4519 , piece_id=G5OUVKDZZJ3SDDD2GO3YXHGE7X5LFQRO75OO6QQRYKPX3UGXN5UQ, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: rpc: dial tcp 104.36.82.47:4519: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.528228282 (58h7m20.373029807s) [noise=true , address=104.36.82.56:4231 , piece_id=2GMQ4EDKXWWHIVHX4ZDNMVKYNY6MZ3TQF5IV6V6RAGRKRSYDJ4JQ, seg_pos={0 0}, upload=55667]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: write tcp 159.223.143.39:38382->104.36.82.56:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:37.528248687 (58h7m20.373050209s) [seg_pos={0 0}, upload=55667]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 21:42:37.563081870 (58h7m20.407883385s) [upload=55667]: Done uploading segments... Jul 26 21:42:37.563106710 (58h7m20.407908233s) [upload=55667]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.038641285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................X.XX...X] duration: 30.028182478s FAIL (2025-07-26 21:42:13 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== :4221 , piece_id=TRRA2ZWGGFM6445C4NMSWYO5H73KTY3OPQH3CM7U55BX74UHQI7Q, seg_pos={0 0}, upload=55658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 159.223.143.39:55862->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Jul 26 21:41:31.132553832 (58h6m13.977355344s) [seg_pos={0 0}, upload=55658]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:41:31.132636072 (58h6m13.977437594s) [upload=55658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:41:31.132642523 (58h6m13.977444046s) [upload=55658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:41:31.189100166 (58h6m14.033901688s) [upload=55658]: Done uploading segments... Jul 26 21:41:31.189118910 (58h6m14.033920432s) [upload=55658]: Upload finished. err: <nil> Jul 26 21:41:43.593978116 (58h6m26.438779633s) [upload=55669]: Starting upload Jul 26 21:41:43.594091278 (58h6m26.438892865s) [upload=55669]: Done starting upload Jul 26 21:41:43.594198711 (58h6m26.439000223s) [upload=55669]: Uploading segments... Jul 26 21:41:43.594617474 (58h6m26.439418997s) [upload=55669]: Got next segment. Inline: false Jul 26 21:41:43.594688129 (58h6m26.439489654s) [upload=55669]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:41:43.594695983 (58h6m26.439497507s) [upload=55669]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:41:43.676662603 (58h6m26.521464134s) [seg_pos={0 0}, upload=55669]: Begin upload segment... Jul 26 21:41:43.676732330 (58h6m26.521533843s) [seg_pos={0 0}, upload=55669]: Done begin upload segment. Jul 26 21:41:43.676761988 (58h6m26.521563521s) [noise=true , address=77.90.31.16:4735 , piece_id=6A3ATQE7FGUR2W6VV72MKVDHFB224PAFUWJHULNQUARZVV47A7CA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.676811055 (58h6m26.521612631s) [noise=true , address=45.91.116.8:4045 , piece_id=V5ZTHGV5XQDLJVA46FGLZKVWRAQHWZANIHML2MM5C5IJXXW6YBGA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.677223825 (58h6m26.522025347s) [noise=true , address=45.91.116.12:4227 , piece_id=QOI37VENLHL74VCVTLSLUXCDQJ5ENHXYREVAYK7NBAMRKLBOTVZQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.677352278 (58h6m26.522153806s) [noise=true , address=45.91.116.8:4047 , piece_id=ONEQP4LIGA33BD6TJPUOKU3RYZUPRMHAYRA6UJFUNA27IODCMEBQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.677584306 (58h6m26.522385831s) [noise=true , address=104.36.82.7:4911 , piece_id=ZL7GXTUTDEHKOAV443PUZF6AZGO2RWVUYKIKDSF5IUQMKENC5YKA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.677836843 (58h6m26.522638374s) [noise=true , address=77.90.31.15:4699 , piece_id=JAKHHUQTJS6PGMPRK4TM34Z6AF4RJ47Q2SXSK2EB724K52M2FPZA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.677907949 (58h6m26.522709481s) [noise=true , address=45.91.116.10:4083 , piece_id=2FLQNVCAOLQJJEFKLKBF7LT76SYYM7NYKJ4CQX6TCDJISNFTJZJA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.678222879 (58h6m26.523024404s) [noise=true , address=104.36.82.15:4259 , piece_id=ATXEGVAJORFE2RZXLEJHCOEUO6QW2AC6LUFLOQ44UVA32JOSEL6A, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.678338388 (58h6m26.52313992s) [noise=true , address=45.91.116.9:4263 , piece_id=TO6JLKYMOWLSFWTQZ5XDUT3MX7EPVMUUWN6XIWRHCJWQYKHTFA2Q, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.678576793 (58h6m26.523378315s) [noise=true , address=104.36.82.13:4186 , piece_id=GEVIXCPVJQ36MZXZ2MFIZ4DTXJVDYNBU5T7JPEX7M2XLIVVHHEQA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.678810262 (58h6m26.52361179s) [noise=true , address=45.91.116.6:4974 , piece_id=6K32JH3TE2AATAPT3B4ZV2W6JLYTV6XZ5BEOZJMPLQZVC5LVNRFA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.678902110 (58h6m26.523703632s) [noise=true , address=45.91.116.14:4190 , piece_id=SNIE3P5ELZZEGRMAWSSG7JW45MRCPJGJIBJP5PLRLPJ4XQ4NVDMQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.679251743 (58h6m26.524053274s) [noise=true , address=77.90.31.19:4842 , piece_id=DR4YEDADRAGGGOB6WBTOW32TY3YMDNVWGC3LVS2HTK3PZ7TYIJXQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.679258304 (58h6m26.524059838s) [noise=true , address=77.90.31.13:4628 , piece_id=RYQ6QPT7CNGA4E74QLISOFLRW6X7YPZUNYUCQRK2UB6OTNFO3FSQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.679716730 (58h6m26.524518264s) [noise=true , address=104.36.82.2:4732 , piece_id=LQCEG6ITTPOS74NFBST4XYUKEAXBX43KGGMLWN4INU3ISZFDN4GA, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.680151303 (58h6m26.524952828s) [noise=true , address=45.91.116.5:4297 , piece_id=GOASWSTM7SNXNV6WTCNXTE6ZFULVZVNTDKB7YNEM7FP6TLWD2SXQ, seg_pos={0 0}, upload=55669]: Uploading piece... Jul 26 21:41:43.756082686 (58h6m26.600884201s) [upload=55669]: Next returned nil segment Jul 26 21:41:43.756098979 (58h6m26.600900492s) [upload=55669]: Waiting for error group managing segments... Jul 26 21:41:43.760489386 (58h6m26.605290901s) [noise=true , address=45.91.116.9:4263 , piece_id=TO6JLKYMOWLSFWTQZ5XDUT3MX7EPVMUUWN6XIWRHCJWQYKHTFA2Q, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.761122019 (58h6m26.605923558s) [noise=true , address=45.91.116.5:4297 , piece_id=GOASWSTM7SNXNV6WTCNXTE6ZFULVZVNTDKB7YNEM7FP6TLWD2SXQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.761589275 (58h6m26.606390791s) [noise=true , address=45.91.116.6:4974 , piece_id=6K32JH3TE2AATAPT3B4ZV2W6JLYTV6XZ5BEOZJMPLQZVC5LVNRFA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.761986968 (58h6m26.606788479s) [noise=true , address=45.91.116.8:4047 , piece_id=ONEQP4LIGA33BD6TJPUOKU3RYZUPRMHAYRA6UJFUNA27IODCMEBQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.762412003 (58h6m26.607213529s) [noise=true , address=45.91.116.14:4190 , piece_id=SNIE3P5ELZZEGRMAWSSG7JW45MRCPJGJIBJP5PLRLPJ4XQ4NVDMQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.762853910 (58h6m26.607655421s) [noise=true , address=45.91.116.12:4227 , piece_id=QOI37VENLHL74VCVTLSLUXCDQJ5ENHXYREVAYK7NBAMRKLBOTVZQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.763284522 (58h6m26.608086048s) [noise=true , address=45.91.116.8:4045 , piece_id=V5ZTHGV5XQDLJVA46FGLZKVWRAQHWZANIHML2MM5C5IJXXW6YBGA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:43.763664187 (58h6m26.608465698s) [noise=true , address=45.91.116.10:4083 , piece_id=2FLQNVCAOLQJJEFKLKBF7LT76SYYM7NYKJ4CQX6TCDJISNFTJZJA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:45.896045410 (58h6m28.740846949s) [noise=true , address=77.90.31.15:4699 , piece_id=JAKHHUQTJS6PGMPRK4TM34Z6AF4RJ47Q2SXSK2EB724K52M2FPZA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:46.907240573 (58h6m29.752042099s) [noise=true , address=77.90.31.16:4735 , piece_id=6A3ATQE7FGUR2W6VV72MKVDHFB224PAFUWJHULNQUARZVV47A7CA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:47.849297242 (58h6m30.694098769s) [noise=true , address=77.90.31.19:4842 , piece_id=DR4YEDADRAGGGOB6WBTOW32TY3YMDNVWGC3LVS2HTK3PZ7TYIJXQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:41:54.264479096 (58h6m37.10928061s) [noise=true , address=77.90.31.13:4628 , piece_id=RYQ6QPT7CNGA4E74QLISOFLRW6X7YPZUNYUCQRK2UB6OTNFO3FSQ, seg_pos={0 0}, upload=55669]: Done uploading piece. err: <nil> Jul 26 21:42:13.556173009 (58h6m56.400974538s) [noise=true , address=104.36.82.2:4732 , piece_id=LQCEG6ITTPOS74NFBST4XYUKEAXBX43KGGMLWN4INU3ISZFDN4GA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:13.556429672 (58h6m56.401231199s) [noise=true , address=104.36.82.15:4259 , piece_id=ATXEGVAJORFE2RZXLEJHCOEUO6QW2AC6LUFLOQ44UVA32JOSEL6A, seg_pos={0 0}, upload=55669]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:13.556505874 (58h6m56.401307399s) [noise=true , address=104.36.82.7:4911 , piece_id=ZL7GXTUTDEHKOAV443PUZF6AZGO2RWVUYKIKDSF5IUQMKENC5YKA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:13.556597462 (58h6m56.40139899s) [noise=true , address=104.36.82.13:4186 , piece_id=GEVIXCPVJQ36MZXZ2MFIZ4DTXJVDYNBU5T7JPEX7M2XLIVVHHEQA, seg_pos={0 0}, upload=55669]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:36276->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:42:13.556673050 (58h6m56.401474584s) [seg_pos={0 0}, upload=55669]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:42:13.581818490 (58h6m56.426620015s) [upload=55669]: Done uploading segments... Jul 26 21:42:13.581886792 (58h6m56.426688319s) [upload=55669]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.028212399s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.......X......] duration: 5.545415106s FAIL (2025-07-26 21:41:42 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.545443566s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 1m0.054622461s FAIL (2025-07-26 21:41:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== ress=79.127.205.235:4687 , piece_id=QUOKWRKEZR3IVYQV3GCFGLJH3Z6HCGJSO3D5JWMBCWA2GGU2RYVQ, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.423534964 (58h5m20.268336497s) [noise=true , address=77.90.31.16:4748 , piece_id=NBRY23WDCMN4GSSJCFBJW7RMSI3IFK22LXTHURLERC37CI74BNHA, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.424421573 (58h5m20.269223104s) [noise=true , address=104.36.82.56:4250 , piece_id=EY5JT5HB4NNW6XCT5NPDC42QSA5XI7UPYLOXUFQL2TW3RIOPGPUA, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.424817215 (58h5m20.269618749s) [noise=true , address=104.36.82.55:4203 , piece_id=MRCPUZI7EOZTUT4XWCWES6BDYSYNI7AI3UY2HV4DQXWLL3OE2RCA, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.425203718 (58h5m20.270005251s) [noise=true , address=104.36.82.38:4707 , piece_id=DO2JZBDO2WU3T4NTX6CNMCINVC3KQAP3JO5DVZPVMAVVYOQ774YQ, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.425564916 (58h5m20.270366452s) [noise=true , address=104.36.82.52:4104 , piece_id=6SSC7QK2WRF4UEFYGAXHKOVDGTVDCBZ6PC4EZFSUDF2NG7AXCKCQ, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.425923321 (58h5m20.270724851s) [noise=true , address=104.36.82.7:4925 , piece_id=ADLG7HYHRUWB67MOKU3Q5E4XUJ4GUEQO37INHSEFYEABUF6ONAPA, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.426279481 (58h5m20.271081019s) [noise=true , address=104.36.82.41:4780 , piece_id=OCJCOZKAPKPTGMLXVNONXR5VD7LP7B7IK3EYXQ6JJBMSLCOIETGQ, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.426646164 (58h5m20.271447701s) [noise=true , address=77.90.31.13:4656 , piece_id=2SMDFGIDI474G3FXOE7DCEHVHVYNSB7BII4VC5CLPDKFNZULWS4Q, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.427096859 (58h5m20.271898393s) [noise=true , address=79.127.205.245:4244 , piece_id=OFWA5W3AYEJRVLAEUGL3PRBXRWYKHJ3KJJG3VZUOXYCMG67S3JMA, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.429523748 (58h5m20.274325282s) [noise=true , address=45.91.116.9:4269 , piece_id=FUDS4OKCKDHUJRDVRLN5EB5MVR6FWPDJGAAYXIXXSTEFMYPFZCIA, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.430206222 (58h5m20.275007757s) [noise=true , address=104.36.82.2:4733 , piece_id=STUZNQNXP7P2EJLDX52HS536E7NDBWNPMSTDPSYDGGVXSGOVOWRQ, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.431155258 (58h5m20.275956792s) [noise=true , address=79.127.201.215:4344 , piece_id=TAVTLOQJKXDNP6DJRUE7CYFDGON7OV3K3XXKO2YAU2JB6FYQCDHQ, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.431610821 (58h5m20.276412359s) [noise=true , address=104.36.82.45:4924 , piece_id=SFHZQUMGBEOWI4VYAX2MQG7AF4MJ5XZWYXXTZFCV5WFALFZ7NKPA, seg_pos={0 0}, upload=55649]: Uploading piece... Jul 26 21:40:37.638653264 (58h5m20.483454794s) [upload=55649]: Next returned nil segment Jul 26 21:40:37.638677731 (58h5m20.483479262s) [upload=55649]: Waiting for error group managing segments... Jul 26 21:40:38.018259259 (58h5m20.863060776s) [noise=true , address=79.127.205.245:4244 , piece_id=OFWA5W3AYEJRVLAEUGL3PRBXRWYKHJ3KJJG3VZUOXYCMG67S3JMA, seg_pos={0 0}, upload=55649]: Done uploading piece. err: <nil> Jul 26 21:40:38.238599500 (58h5m21.083401016s) [noise=true , address=79.127.205.235:4687 , piece_id=QUOKWRKEZR3IVYQV3GCFGLJH3Z6HCGJSO3D5JWMBCWA2GGU2RYVQ, seg_pos={0 0}, upload=55649]: Done uploading piece. err: <nil> Jul 26 21:40:39.613658713 (58h5m22.458460235s) [noise=true , address=77.90.31.13:4656 , piece_id=2SMDFGIDI474G3FXOE7DCEHVHVYNSB7BII4VC5CLPDKFNZULWS4Q, seg_pos={0 0}, upload=55649]: Done uploading piece. err: <nil> Jul 26 21:40:40.304848128 (58h5m23.149649653s) [noise=true , address=77.90.31.16:4748 , piece_id=NBRY23WDCMN4GSSJCFBJW7RMSI3IFK22LXTHURLERC37CI74BNHA, seg_pos={0 0}, upload=55649]: Done uploading piece. err: <nil> Jul 26 21:40:40.905789201 (58h5m23.750590717s) [noise=true , address=77.90.31.15:4714 , piece_id=ZXP26VI7IHL2WENDZHNUBU6SGOFDGO35WTH4ESUJCQHRKFMZDC5A, seg_pos={0 0}, upload=55649]: Done uploading piece. err: <nil> Jul 26 21:40:41.085738333 (58h5m23.930539856s) [noise=true , address=79.127.201.215:4344 , piece_id=TAVTLOQJKXDNP6DJRUE7CYFDGON7OV3K3XXKO2YAU2JB6FYQCDHQ, seg_pos={0 0}, upload=55649]: Done uploading piece. err: <nil> Jul 26 21:40:42.625342222 (58h5m25.47014375s) [noise=true , address=45.91.116.9:4269 , piece_id=FUDS4OKCKDHUJRDVRLN5EB5MVR6FWPDJGAAYXIXXSTEFMYPFZCIA, seg_pos={0 0}, upload=55649]: Done uploading piece. err: <nil> Jul 26 21:41:37.365268323 (58h6m20.210069846s) [noise=true , address=104.36.82.54:4179 , piece_id=HOZUIMFSQLDQGFSBZVMZZ25JR3JICA5A2HP4R22RTIAXSBFOYLDA, seg_pos={0 0}, upload=55649]: Done uploading piece. err: ecclient: upload failed (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP, address:104.36.82.54:4179): protocol: rpc: dial tcp 104.36.82.54:4179: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:41:37.365490105 (58h6m20.210291627s) [noise=true , address=104.36.82.38:4707 , piece_id=DO2JZBDO2WU3T4NTX6CNMCINVC3KQAP3JO5DVZPVMAVVYOQ774YQ, seg_pos={0 0}, upload=55649]: Done uploading piece. err: ecclient: upload failed (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T, address:104.36.82.38:4707): protocol: write tcp 159.223.143.39:46996->104.36.82.38:4707: use of closed network connection; write tcp 159.223.143.39:46996->104.36.82.38:4707: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46996->104.36.82.38:4707: use of closed network connection Jul 26 21:41:37.365698631 (58h6m20.210500158s) [noise=true , address=104.36.82.7:4925 , piece_id=ADLG7HYHRUWB67MOKU3Q5E4XUJ4GUEQO37INHSEFYEABUF6ONAPA, seg_pos={0 0}, upload=55649]: Done uploading piece. err: ecclient: upload failed (node:1RermE4TG8CP8Z7yeAMXDbLaztghzKSpCJQFjDvLCCsy4y3rdn, address:104.36.82.7:4925): protocol: write tcp 159.223.143.39:38444->104.36.82.7:4925: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:41:37.369028715 (58h6m20.213830237s) [noise=true , address=104.36.82.46:4502 , piece_id=DS6XV6YB6INPOMZMAFGDTNVSMCLI2QKBPKVRWISCWEXL4A2AGRGA, seg_pos={0 0}, upload=55649]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:41:37.369136280 (58h6m20.213937805s) [noise=true , address=104.36.82.53:4148 , piece_id=QGS4SZ66LM53FYGSXLGAAIOTJUJGCK65FQHIHROPFN6UVOR4N7IQ, seg_pos={0 0}, upload=55649]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: rpc: dial tcp 104.36.82.53:4148: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:41:37.369242748 (58h6m20.214044269s) [noise=true , address=104.36.82.41:4780 , piece_id=OCJCOZKAPKPTGMLXVNONXR5VD7LP7B7IK3EYXQ6JJBMSLCOIETGQ, seg_pos={0 0}, upload=55649]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:104.36.82.41:4780): protocol: rpc: dial tcp 104.36.82.41:4780: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:41:37.369316313 (58h6m20.214117835s) [noise=true , address=104.36.82.2:4733 , piece_id=STUZNQNXP7P2EJLDX52HS536E7NDBWNPMSTDPSYDGGVXSGOVOWRQ, seg_pos={0 0}, upload=55649]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:104.36.82.2:4733): protocol: rpc: dial tcp 104.36.82.2:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:41:37.369706836 (58h6m20.214508358s) [noise=true , address=104.36.82.56:4250 , piece_id=EY5JT5HB4NNW6XCT5NPDC42QSA5XI7UPYLOXUFQL2TW3RIOPGPUA, seg_pos={0 0}, upload=55649]: Done uploading piece. err: ecclient: upload failed (node:12oHdA7jYDKSYwWLB975bJ3Fu27xjfZDC9utpnQpszmYjVz2CdD, address:104.36.82.56:4250): protocol: rpc: dial tcp 104.36.82.56:4250: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:41:37.369823720 (58h6m20.214625241s) [noise=true , address=104.36.82.55:4203 , piece_id=MRCPUZI7EOZTUT4XWCWES6BDYSYNI7AI3UY2HV4DQXWLL3OE2RCA, seg_pos={0 0}, upload=55649]: Done uploading piece. err: ecclient: upload failed (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU, address:104.36.82.55:4203): protocol: rpc: dial tcp 104.36.82.55:4203: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:41:37.369897543 (58h6m20.214699066s) [noise=true , address=104.36.82.45:4924 , piece_id=SFHZQUMGBEOWI4VYAX2MQG7AF4MJ5XZWYXXTZFCV5WFALFZ7NKPA, seg_pos={0 0}, upload=55649]: Done uploading piece. err: ecclient: upload failed (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs, address:104.36.82.45:4924): protocol: rpc: dial tcp 104.36.82.45:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:41:37.370264465 (58h6m20.215065989s) [noise=true , address=104.36.82.52:4104 , piece_id=6SSC7QK2WRF4UEFYGAXHKOVDGTVDCBZ6PC4EZFSUDF2NG7AXCKCQ, seg_pos={0 0}, upload=55649]: Done uploading piece. err: ecclient: upload failed (node:12qmPpCmXAvuJZn2KR5Kr4nHYwKEzWvwqZ82nqNka5u1mKaZmXs, address:104.36.82.52:4104): protocol: write tcp 159.223.143.39:39708->104.36.82.52:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:41:37.370289626 (58h6m20.21509114s) [seg_pos={0 0}, upload=55649]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 21:41:37.404672260 (58h6m20.249473787s) [upload=55649]: Done uploading segments... Jul 26 21:41:37.404698934 (58h6m20.249500458s) [upload=55649]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.054654737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X.XX...] duration: 18.987330092s FAIL (2025-07-26 21:41:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== l: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:04.071127088 (58h2m46.9159286s) [noise=true , address=104.36.82.2:4730 , piece_id=DCDACT3346A3RIH2TEVDOUPKON23MF2RVCPI7QOX6DOKK4ZNJPVQ, seg_pos={0 0}, upload=55595]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:04.071172355 (58h2m46.915973887s) [seg_pos={0 0}, upload=55595]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:38:04.097485648 (58h2m46.942287179s) [upload=55595]: Done uploading segments... Jul 26 21:38:04.097515880 (58h2m46.94231741s) [upload=55595]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 21:38:04.253038561 (58h2m47.097840122s) [upload=55604]: Starting upload Jul 26 21:38:04.253119101 (58h2m47.097920632s) [upload=55604]: Done starting upload Jul 26 21:38:04.253491465 (58h2m47.098292991s) [upload=55604]: Uploading segments... Jul 26 21:38:04.253557592 (58h2m47.098359128s) [upload=55604]: Got next segment. Inline: false Jul 26 21:38:04.253602720 (58h2m47.098404255s) [upload=55604]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:38:04.253608801 (58h2m47.098410338s) [upload=55604]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:38:04.361137816 (58h2m47.205939345s) [seg_pos={0 0}, upload=55604]: Begin upload segment... Jul 26 21:38:04.361201367 (58h2m47.20600288s) [seg_pos={0 0}, upload=55604]: Done begin upload segment. Jul 26 21:38:04.361228042 (58h2m47.206029566s) [noise=true , address=45.91.116.4:4938 , piece_id=Y2CLY5IUVMOI2YFPVJHWJ42Y2XAU4W2NHEC76VSP2II27XFZ6BKQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.361319061 (58h2m47.206120595s) [noise=true , address=45.91.116.13:4153 , piece_id=HBLDRHKMQ5NGLQGEHJ3FDB2BSZKX7OQEOMW4BYLUUYYB5YWEW4UQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.361841399 (58h2m47.206642931s) [noise=true , address=45.91.116.5:4299 , piece_id=3PCIKAYO3KPV57KCJ76PAXB5A2BIK5OESLK2RWKZ2C6OR6CVT5PQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.361860297 (58h2m47.206661882s) [noise=true , address=45.91.116.9:4263 , piece_id=Z33MMFDV7RLFAFHEGW2DW6VMQYRCBOHNVMJTHQ3YRUGWMZ4VQEGQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.362207262 (58h2m47.20700879s) [noise=true , address=45.91.116.12:4225 , piece_id=IZTBG7W52VI2URXM7U7NQGCPVO2N52PK76YUNQEFJ3M5CL6ZMBCQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.362328935 (58h2m47.207130461s) [noise=true , address=45.91.116.6:4973 , piece_id=Q475ATN54FHHJLMVRWWP5QWBS3HYLTJ6SSYQXZKO7JRDV23CNKUA, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.362588847 (58h2m47.207390376s) [noise=true , address=104.36.82.7:4909 , piece_id=LMAXLVRYYMXOGAYQCDZX3SQZHVQTG6HAERY2QZMCBUB3CJTEYRIQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.362800269 (58h2m47.207601806s) [noise=true , address=104.36.82.5:4839 , piece_id=7NM4RSRRUWDR2NFDYPXD2YICAHV3A53CQMNE4YIO75OGB3W77E3Q, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.362925173 (58h2m47.207726718s) [noise=true , address=45.91.116.5:4297 , piece_id=SG4CGBIJKMTGGDNBHWMMGYLLZEYLODZR2MCCURF7U6CR3JTW2OVQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.363344622 (58h2m47.208146149s) [noise=true , address=77.90.31.10:4518 , piece_id=TYQFS2Z2Q3KNQS2UJTZV4KEMMWNP7HUFVQ7NPGNZGXTXJFRUC4WQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.363397001 (58h2m47.208198525s) [noise=true , address=45.91.116.7:4011 , piece_id=VYZGBHB6Q5GAU7EZWI5OAY5MLAW6QHXJRZVO75L7G4TCE73VVHVA, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.363775838 (58h2m47.208577374s) [noise=true , address=104.36.82.2:4730 , piece_id=EKAYL22UH5FP2RDRGJS6DNGWGANMC6OWXMSVSUJI7BSDQNAI6JDQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.363832710 (58h2m47.208634234s) [noise=true , address=45.91.116.7:4010 , piece_id=UFJLHO7NCDEJH2HTZWRS7EYR62ACTX5MGFZW6LJAUS47ZBLSCICQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.364387541 (58h2m47.209189077s) [noise=true , address=45.91.116.9:4262 , piece_id=KL46QK22IJDAORPSVTO4KZ7M2NY6OBIPZGSR64QOMJKAISYHQUCA, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.364850896 (58h2m47.209652429s) [noise=true , address=45.91.116.11:4118 , piece_id=CF2HMEFVAQV2LRJBENTIOPHQG5GPO5UDSTCQTJSOIZ52VI7PQAIA, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.365310687 (58h2m47.210112216s) [noise=true , address=104.36.82.10:4079 , piece_id=N7DAQEUB52IC5RDDMURKJNY7GHZPEPSLLPEQJNMZMDCXLTIDQ5VA, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.486260172 (58h2m47.331061701s) [upload=55604]: Next returned nil segment Jul 26 21:38:04.486281865 (58h2m47.331083399s) [upload=55604]: Waiting for error group managing segments... Jul 26 21:38:04.490565046 (58h2m47.335366573s) [noise=true , address=45.91.116.12:4225 , piece_id=IZTBG7W52VI2URXM7U7NQGCPVO2N52PK76YUNQEFJ3M5CL6ZMBCQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.492835447 (58h2m47.337636956s) [noise=true , address=45.91.116.4:4938 , piece_id=Y2CLY5IUVMOI2YFPVJHWJ42Y2XAU4W2NHEC76VSP2II27XFZ6BKQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.493455836 (58h2m47.33825736s) [noise=true , address=45.91.116.13:4153 , piece_id=HBLDRHKMQ5NGLQGEHJ3FDB2BSZKX7OQEOMW4BYLUUYYB5YWEW4UQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.494061675 (58h2m47.338863196s) [noise=true , address=45.91.116.7:4010 , piece_id=UFJLHO7NCDEJH2HTZWRS7EYR62ACTX5MGFZW6LJAUS47ZBLSCICQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.494647346 (58h2m47.339448872s) [noise=true , address=45.91.116.6:4973 , piece_id=Q475ATN54FHHJLMVRWWP5QWBS3HYLTJ6SSYQXZKO7JRDV23CNKUA, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.495242552 (58h2m47.340044073s) [noise=true , address=45.91.116.5:4299 , piece_id=3PCIKAYO3KPV57KCJ76PAXB5A2BIK5OESLK2RWKZ2C6OR6CVT5PQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.495932061 (58h2m47.340733571s) [noise=true , address=45.91.116.9:4263 , piece_id=Z33MMFDV7RLFAFHEGW2DW6VMQYRCBOHNVMJTHQ3YRUGWMZ4VQEGQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.496696157 (58h2m47.34149767s) [noise=true , address=45.91.116.9:4262 , piece_id=KL46QK22IJDAORPSVTO4KZ7M2NY6OBIPZGSR64QOMJKAISYHQUCA, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.497481590 (58h2m47.342283116s) [noise=true , address=45.91.116.11:4118 , piece_id=CF2HMEFVAQV2LRJBENTIOPHQG5GPO5UDSTCQTJSOIZ52VI7PQAIA, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.498063319 (58h2m47.342864847s) [noise=true , address=45.91.116.7:4011 , piece_id=VYZGBHB6Q5GAU7EZWI5OAY5MLAW6QHXJRZVO75L7G4TCE73VVHVA, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.498614972 (58h2m47.343416502s) [noise=true , address=45.91.116.5:4297 , piece_id=SG4CGBIJKMTGGDNBHWMMGYLLZEYLODZR2MCCURF7U6CR3JTW2OVQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:06.393367299 (58h2m49.238168827s) [noise=true , address=77.90.31.10:4518 , piece_id=TYQFS2Z2Q3KNQS2UJTZV4KEMMWNP7HUFVQ7NPGNZGXTXJFRUC4WQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:34.221526535 (58h3m17.066328071s) [noise=true , address=104.36.82.7:4909 , piece_id=LMAXLVRYYMXOGAYQCDZX3SQZHVQTG6HAERY2QZMCBUB3CJTEYRIQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:34.221722111 (58h3m17.066523634s) [noise=true , address=104.36.82.5:4839 , piece_id=7NM4RSRRUWDR2NFDYPXD2YICAHV3A53CQMNE4YIO75OGB3W77E3Q, seg_pos={0 0}, upload=55604]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:54146->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:34.221742641 (58h3m17.066544228s) [noise=true , address=104.36.82.2:4730 , piece_id=EKAYL22UH5FP2RDRGJS6DNGWGANMC6OWXMSVSUJI7BSDQNAI6JDQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:34.221830554 (58h3m17.066632082s) [noise=true , address=104.36.82.10:4079 , piece_id=N7DAQEUB52IC5RDDMURKJNY7GHZPEPSLLPEQJNMZMDCXLTIDQ5VA, seg_pos={0 0}, upload=55604]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58764->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:34.221900476 (58h3m17.06670201s) [seg_pos={0 0}, upload=55604]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:38:34.251293801 (58h3m17.096095357s) [upload=55604]: Done uploading segments... Jul 26 21:38:34.251329525 (58h3m17.096131051s) [upload=55604]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 18.987382401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.......X.....] duration: 3.089077255s FAIL (2025-07-26 21:41:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 3.089124341s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 1m0.030096587s FAIL (2025-07-26 21:40:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Z2DN56WGFPK2XNHSECGJ7OQE2AMXBX7MP55XZAC7UPDGA, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.342745927 (58h4m20.187547459s) [noise=true , address=79.127.205.246:4266 , piece_id=J3BKYKJIP37BRUKGLLWAKUFSBNQUIVRC2QNKTOUSVLA5BJLLIMGQ, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.343244644 (58h4m20.188046344s) [noise=true , address=79.127.201.215:4352 , piece_id=3T4ONDY6ZZY3BZDAASNDSYZP6W2UCZCFYWIDB4HP54OJ3X5IKZNA, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.343730029 (58h4m20.188531561s) [noise=true , address=104.36.82.20:4465 , piece_id=QUNO3PQFFOS4LXRQXHF4PGRZVVTTKCGFYQTD6L7AP4INJZ34JDKA, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.344269137 (58h4m20.189070671s) [noise=true , address=104.36.82.21:4499 , piece_id=FDWNMIHEFYKEG7EE2SLSERBUVTU3546FU37NFVMFTAWPLV2BVBBQ, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.344765442 (58h4m20.189566997s) [noise=true , address=77.90.31.16:4766 , piece_id=5T7THOY2QXGOMXJQDAXRCNB7N7NN7HCGCG4I47VMG5HWCTI565MQ, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.345249762 (58h4m20.190051295s) [noise=true , address=104.36.82.55:4224 , piece_id=TUEZISZ5D5PUPWOK7R64B67OSFB72DAHCBAYX32MDF25AGH4NZMA, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.345726117 (58h4m20.190527651s) [noise=true , address=104.36.82.52:4108 , piece_id=4ANZRGVNLLIMSRZO4X4ENFSQ6QHUBQCPIUUWDJBSJMHNGZ2YQJKQ, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.346192025 (58h4m20.190993555s) [noise=true , address=104.36.82.47:4523 , piece_id=H27GEJ4GBMPDJGJDNQKPAKYNHIUASIR6Y54L7BMYTQLPENZWFQHA, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.346687327 (58h4m20.191488859s) [noise=true , address=104.36.82.43:4850 , piece_id=E77P5TLPELOZOPNLU3OSKVT7VF3K4FAGMWKFH7XFMSN72NRFE3HA, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.347229697 (58h4m20.192031232s) [noise=true , address=104.36.82.53:4139 , piece_id=HFY3KRRG42FTOKQA4EHARNE7JG63CAZB2APSED2PKTQB6WCX2VSQ, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.347705659 (58h4m20.192507192s) [noise=true , address=79.127.219.38:4015 , piece_id=MVBGPIZLILYVE6YHLDF4MS6LVIGIFYAUXA4TM4GTC7ZKWFM6XL5A, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.348150837 (58h4m20.192952369s) [noise=true , address=104.36.82.51:4073 , piece_id=URKHMOF6TGSUDD2ZZ5BR2NKW7BIE37KG4OSENHYRMK5JHX2BNIGQ, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.348683662 (58h4m20.193485195s) [noise=true , address=79.127.205.235:4676 , piece_id=LCGXSOB7RQ4UWEV6KYRVCQZL56YUFJUKAE32ZT4SBH4MNHBXIBYA, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.349256518 (58h4m20.194058054s) [noise=true , address=104.36.82.54:4158 , piece_id=A3DNAF6ANMRBHHOE7J4CFKLX7VR4X62M4HDOG2GYDFU3CA4G3WZA, seg_pos={0 0}, upload=55632]: Uploading piece... Jul 26 21:39:37.405537112 (58h4m20.250338636s) [upload=55632]: Next returned nil segment Jul 26 21:39:37.405557564 (58h4m20.250359099s) [upload=55632]: Waiting for error group managing segments... Jul 26 21:39:37.418396654 (58h4m20.263198165s) [noise=true , address=79.127.219.38:4015 , piece_id=MVBGPIZLILYVE6YHLDF4MS6LVIGIFYAUXA4TM4GTC7ZKWFM6XL5A, seg_pos={0 0}, upload=55632]: Done uploading piece. err: <nil> Jul 26 21:39:37.524776475 (58h4m20.369577985s) [noise=true , address=104.36.82.20:4465 , piece_id=QUNO3PQFFOS4LXRQXHF4PGRZVVTTKCGFYQTD6L7AP4INJZ34JDKA, seg_pos={0 0}, upload=55632]: Done uploading piece. err: <nil> Jul 26 21:39:37.971102136 (58h4m20.815903654s) [noise=true , address=79.127.205.235:4676 , piece_id=LCGXSOB7RQ4UWEV6KYRVCQZL56YUFJUKAE32ZT4SBH4MNHBXIBYA, seg_pos={0 0}, upload=55632]: Done uploading piece. err: <nil> Jul 26 21:39:38.463791626 (58h4m21.308593143s) [noise=true , address=79.127.205.246:4266 , piece_id=J3BKYKJIP37BRUKGLLWAKUFSBNQUIVRC2QNKTOUSVLA5BJLLIMGQ, seg_pos={0 0}, upload=55632]: Done uploading piece. err: <nil> Jul 26 21:39:39.262469356 (58h4m22.107270879s) [noise=true , address=79.127.201.215:4352 , piece_id=3T4ONDY6ZZY3BZDAASNDSYZP6W2UCZCFYWIDB4HP54OJ3X5IKZNA, seg_pos={0 0}, upload=55632]: Done uploading piece. err: <nil> Jul 26 21:39:39.615827723 (58h4m22.46062924s) [noise=true , address=79.127.219.51:4212 , piece_id=QTCEQDVNSKUYX7FSIPLABBWLJEZW3V5S37CTP7WLBPU4TYBQDKDQ, seg_pos={0 0}, upload=55632]: Done uploading piece. err: <nil> Jul 26 21:39:44.025377493 (58h4m26.870179017s) [noise=true , address=77.90.31.16:4766 , piece_id=5T7THOY2QXGOMXJQDAXRCNB7N7NN7HCGCG4I47VMG5HWCTI565MQ, seg_pos={0 0}, upload=55632]: Done uploading piece. err: <nil> Jul 26 21:40:37.257465007 (58h5m20.102266536s) [noise=true , address=104.36.82.52:4108 , piece_id=4ANZRGVNLLIMSRZO4X4ENFSQ6QHUBQCPIUUWDJBSJMHNGZ2YQJKQ, seg_pos={0 0}, upload=55632]: Done uploading piece. err: ecclient: upload failed (node:12XJJZCUSHUS52YC6jhoPB4ayjsyKZT477PymPj4inkZtdUvheT, address:104.36.82.52:4108): protocol: rpc: dial tcp 104.36.82.52:4108: i/o timeout; conn closed: op Jul 26 21:40:37.257494936 (58h5m20.102296461s) [noise=true , address=104.36.82.46:4503 , piece_id=CITZGJJZ2DN56WGFPK2XNHSECGJ7OQE2AMXBX7MP55XZAC7UPDGA, seg_pos={0 0}, upload=55632]: Done uploading piece. err: ecclient: upload failed (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm, address:104.36.82.46:4503): protocol: write tcp 159.223.143.39:60302->104.36.82.46:4503: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:40:37.257575504 (58h5m20.102377026s) [noise=true , address=104.36.82.51:4073 , piece_id=URKHMOF6TGSUDD2ZZ5BR2NKW7BIE37KG4OSENHYRMK5JHX2BNIGQ, seg_pos={0 0}, upload=55632]: Done uploading piece. err: ecclient: upload failed (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK, address:104.36.82.51:4073): protocol: write tcp 159.223.143.39:45948->104.36.82.51:4073: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:40:37.257620788 (58h5m20.102422312s) [noise=true , address=104.36.82.55:4224 , piece_id=TUEZISZ5D5PUPWOK7R64B67OSFB72DAHCBAYX32MDF25AGH4NZMA, seg_pos={0 0}, upload=55632]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:40:37.257744589 (58h5m20.102546112s) [noise=true , address=104.36.82.49:4004 , piece_id=SYSXCVT3VD7AXCG5H5A3CX45NJWN5BZ5P7T45QFWNU4E2W3RBMJQ, seg_pos={0 0}, upload=55632]: Done uploading piece. err: ecclient: upload failed (node:1iSAPoDgvJmcbXpHKpVkpNHzso3rvFX5PJa8PooBmohS3BgGKZ, address:104.36.82.49:4004): protocol: write tcp 159.223.143.39:47490->104.36.82.49:4004: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:40:37.257803848 (58h5m20.102605371s) [noise=true , address=104.36.82.43:4850 , piece_id=E77P5TLPELOZOPNLU3OSKVT7VF3K4FAGMWKFH7XFMSN72NRFE3HA, seg_pos={0 0}, upload=55632]: Done uploading piece. err: ecclient: upload failed (node:1CCMafkZzgzS3tBh9SDoTS6tx1hNUfWqH56ebYzxLpcVnK5Cix, address:104.36.82.43:4850): protocol: rpc: dial tcp 104.36.82.43:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:40:37.257935258 (58h5m20.102736785s) [noise=true , address=104.36.82.53:4139 , piece_id=HFY3KRRG42FTOKQA4EHARNE7JG63CAZB2APSED2PKTQB6WCX2VSQ, seg_pos={0 0}, upload=55632]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: rpc: dial tcp 104.36.82.53:4139: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:40:37.258069562 (58h5m20.102871088s) [noise=true , address=104.36.82.56:4256 , piece_id=HY7CT5UFXA2RKSK2QD3XNQEMRSOSNS2FRFKYG3BQ6XUFHGR352OQ, seg_pos={0 0}, upload=55632]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: write tcp 159.223.143.39:36214->104.36.82.56:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:40:37.258201459 (58h5m20.103002982s) [noise=true , address=104.36.82.54:4158 , piece_id=A3DNAF6ANMRBHHOE7J4CFKLX7VR4X62M4HDOG2GYDFU3CA4G3WZA, seg_pos={0 0}, upload=55632]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:104.36.82.54:4158): protocol: rpc: dial tcp 104.36.82.54:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:40:37.258236694 (58h5m20.10303822s) [noise=true , address=104.36.82.21:4499 , piece_id=FDWNMIHEFYKEG7EE2SLSERBUVTU3546FU37NFVMFTAWPLV2BVBBQ, seg_pos={0 0}, upload=55632]: Done uploading piece. err: ecclient: upload failed (node:1218y8DtJNTTu94dbqGhywmHR4s99zcz9issmMmpgwusr2KgZxc, address:104.36.82.21:4499): protocol: write tcp 159.223.143.39:37580->104.36.82.21:4499: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:40:37.258288188 (58h5m20.103089716s) [noise=true , address=104.36.82.47:4523 , piece_id=H27GEJ4GBMPDJGJDNQKPAKYNHIUASIR6Y54L7BMYTQLPENZWFQHA, seg_pos={0 0}, upload=55632]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: rpc: dial tcp 104.36.82.47:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:40:37.258329613 (58h5m20.103131147s) [seg_pos={0 0}, upload=55632]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 21:40:37.285041951 (58h5m20.129843475s) [upload=55632]: Done uploading segments... Jul 26 21:40:37.285069496 (58h5m20.129871017s) [upload=55632]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.030123064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X.XX..] duration: 15.335603498s FAIL (2025-07-26 21:40:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== l: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:04.071127088 (58h2m46.9159286s) [noise=true , address=104.36.82.2:4730 , piece_id=DCDACT3346A3RIH2TEVDOUPKON23MF2RVCPI7QOX6DOKK4ZNJPVQ, seg_pos={0 0}, upload=55595]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:04.071172355 (58h2m46.915973887s) [seg_pos={0 0}, upload=55595]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:38:04.097485648 (58h2m46.942287179s) [upload=55595]: Done uploading segments... Jul 26 21:38:04.097515880 (58h2m46.94231741s) [upload=55595]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 21:38:04.253038561 (58h2m47.097840122s) [upload=55604]: Starting upload Jul 26 21:38:04.253119101 (58h2m47.097920632s) [upload=55604]: Done starting upload Jul 26 21:38:04.253491465 (58h2m47.098292991s) [upload=55604]: Uploading segments... Jul 26 21:38:04.253557592 (58h2m47.098359128s) [upload=55604]: Got next segment. Inline: false Jul 26 21:38:04.253602720 (58h2m47.098404255s) [upload=55604]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:38:04.253608801 (58h2m47.098410338s) [upload=55604]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:38:04.361137816 (58h2m47.205939345s) [seg_pos={0 0}, upload=55604]: Begin upload segment... Jul 26 21:38:04.361201367 (58h2m47.20600288s) [seg_pos={0 0}, upload=55604]: Done begin upload segment. Jul 26 21:38:04.361228042 (58h2m47.206029566s) [noise=true , address=45.91.116.4:4938 , piece_id=Y2CLY5IUVMOI2YFPVJHWJ42Y2XAU4W2NHEC76VSP2II27XFZ6BKQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.361319061 (58h2m47.206120595s) [noise=true , address=45.91.116.13:4153 , piece_id=HBLDRHKMQ5NGLQGEHJ3FDB2BSZKX7OQEOMW4BYLUUYYB5YWEW4UQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.361841399 (58h2m47.206642931s) [noise=true , address=45.91.116.5:4299 , piece_id=3PCIKAYO3KPV57KCJ76PAXB5A2BIK5OESLK2RWKZ2C6OR6CVT5PQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.361860297 (58h2m47.206661882s) [noise=true , address=45.91.116.9:4263 , piece_id=Z33MMFDV7RLFAFHEGW2DW6VMQYRCBOHNVMJTHQ3YRUGWMZ4VQEGQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.362207262 (58h2m47.20700879s) [noise=true , address=45.91.116.12:4225 , piece_id=IZTBG7W52VI2URXM7U7NQGCPVO2N52PK76YUNQEFJ3M5CL6ZMBCQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.362328935 (58h2m47.207130461s) [noise=true , address=45.91.116.6:4973 , piece_id=Q475ATN54FHHJLMVRWWP5QWBS3HYLTJ6SSYQXZKO7JRDV23CNKUA, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.362588847 (58h2m47.207390376s) [noise=true , address=104.36.82.7:4909 , piece_id=LMAXLVRYYMXOGAYQCDZX3SQZHVQTG6HAERY2QZMCBUB3CJTEYRIQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.362800269 (58h2m47.207601806s) [noise=true , address=104.36.82.5:4839 , piece_id=7NM4RSRRUWDR2NFDYPXD2YICAHV3A53CQMNE4YIO75OGB3W77E3Q, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.362925173 (58h2m47.207726718s) [noise=true , address=45.91.116.5:4297 , piece_id=SG4CGBIJKMTGGDNBHWMMGYLLZEYLODZR2MCCURF7U6CR3JTW2OVQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.363344622 (58h2m47.208146149s) [noise=true , address=77.90.31.10:4518 , piece_id=TYQFS2Z2Q3KNQS2UJTZV4KEMMWNP7HUFVQ7NPGNZGXTXJFRUC4WQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.363397001 (58h2m47.208198525s) [noise=true , address=45.91.116.7:4011 , piece_id=VYZGBHB6Q5GAU7EZWI5OAY5MLAW6QHXJRZVO75L7G4TCE73VVHVA, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.363775838 (58h2m47.208577374s) [noise=true , address=104.36.82.2:4730 , piece_id=EKAYL22UH5FP2RDRGJS6DNGWGANMC6OWXMSVSUJI7BSDQNAI6JDQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.363832710 (58h2m47.208634234s) [noise=true , address=45.91.116.7:4010 , piece_id=UFJLHO7NCDEJH2HTZWRS7EYR62ACTX5MGFZW6LJAUS47ZBLSCICQ, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.364387541 (58h2m47.209189077s) [noise=true , address=45.91.116.9:4262 , piece_id=KL46QK22IJDAORPSVTO4KZ7M2NY6OBIPZGSR64QOMJKAISYHQUCA, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.364850896 (58h2m47.209652429s) [noise=true , address=45.91.116.11:4118 , piece_id=CF2HMEFVAQV2LRJBENTIOPHQG5GPO5UDSTCQTJSOIZ52VI7PQAIA, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.365310687 (58h2m47.210112216s) [noise=true , address=104.36.82.10:4079 , piece_id=N7DAQEUB52IC5RDDMURKJNY7GHZPEPSLLPEQJNMZMDCXLTIDQ5VA, seg_pos={0 0}, upload=55604]: Uploading piece... Jul 26 21:38:04.486260172 (58h2m47.331061701s) [upload=55604]: Next returned nil segment Jul 26 21:38:04.486281865 (58h2m47.331083399s) [upload=55604]: Waiting for error group managing segments... Jul 26 21:38:04.490565046 (58h2m47.335366573s) [noise=true , address=45.91.116.12:4225 , piece_id=IZTBG7W52VI2URXM7U7NQGCPVO2N52PK76YUNQEFJ3M5CL6ZMBCQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.492835447 (58h2m47.337636956s) [noise=true , address=45.91.116.4:4938 , piece_id=Y2CLY5IUVMOI2YFPVJHWJ42Y2XAU4W2NHEC76VSP2II27XFZ6BKQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.493455836 (58h2m47.33825736s) [noise=true , address=45.91.116.13:4153 , piece_id=HBLDRHKMQ5NGLQGEHJ3FDB2BSZKX7OQEOMW4BYLUUYYB5YWEW4UQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.494061675 (58h2m47.338863196s) [noise=true , address=45.91.116.7:4010 , piece_id=UFJLHO7NCDEJH2HTZWRS7EYR62ACTX5MGFZW6LJAUS47ZBLSCICQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.494647346 (58h2m47.339448872s) [noise=true , address=45.91.116.6:4973 , piece_id=Q475ATN54FHHJLMVRWWP5QWBS3HYLTJ6SSYQXZKO7JRDV23CNKUA, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.495242552 (58h2m47.340044073s) [noise=true , address=45.91.116.5:4299 , piece_id=3PCIKAYO3KPV57KCJ76PAXB5A2BIK5OESLK2RWKZ2C6OR6CVT5PQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.495932061 (58h2m47.340733571s) [noise=true , address=45.91.116.9:4263 , piece_id=Z33MMFDV7RLFAFHEGW2DW6VMQYRCBOHNVMJTHQ3YRUGWMZ4VQEGQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.496696157 (58h2m47.34149767s) [noise=true , address=45.91.116.9:4262 , piece_id=KL46QK22IJDAORPSVTO4KZ7M2NY6OBIPZGSR64QOMJKAISYHQUCA, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.497481590 (58h2m47.342283116s) [noise=true , address=45.91.116.11:4118 , piece_id=CF2HMEFVAQV2LRJBENTIOPHQG5GPO5UDSTCQTJSOIZ52VI7PQAIA, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.498063319 (58h2m47.342864847s) [noise=true , address=45.91.116.7:4011 , piece_id=VYZGBHB6Q5GAU7EZWI5OAY5MLAW6QHXJRZVO75L7G4TCE73VVHVA, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:04.498614972 (58h2m47.343416502s) [noise=true , address=45.91.116.5:4297 , piece_id=SG4CGBIJKMTGGDNBHWMMGYLLZEYLODZR2MCCURF7U6CR3JTW2OVQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:06.393367299 (58h2m49.238168827s) [noise=true , address=77.90.31.10:4518 , piece_id=TYQFS2Z2Q3KNQS2UJTZV4KEMMWNP7HUFVQ7NPGNZGXTXJFRUC4WQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: <nil> Jul 26 21:38:34.221526535 (58h3m17.066328071s) [noise=true , address=104.36.82.7:4909 , piece_id=LMAXLVRYYMXOGAYQCDZX3SQZHVQTG6HAERY2QZMCBUB3CJTEYRIQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:34.221722111 (58h3m17.066523634s) [noise=true , address=104.36.82.5:4839 , piece_id=7NM4RSRRUWDR2NFDYPXD2YICAHV3A53CQMNE4YIO75OGB3W77E3Q, seg_pos={0 0}, upload=55604]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:54146->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:34.221742641 (58h3m17.066544228s) [noise=true , address=104.36.82.2:4730 , piece_id=EKAYL22UH5FP2RDRGJS6DNGWGANMC6OWXMSVSUJI7BSDQNAI6JDQ, seg_pos={0 0}, upload=55604]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:34.221830554 (58h3m17.066632082s) [noise=true , address=104.36.82.10:4079 , piece_id=N7DAQEUB52IC5RDDMURKJNY7GHZPEPSLLPEQJNMZMDCXLTIDQ5VA, seg_pos={0 0}, upload=55604]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:58764->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:38:34.221900476 (58h3m17.06670201s) [seg_pos={0 0}, upload=55604]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:38:34.251293801 (58h3m17.096095357s) [upload=55604]: Done uploading segments... Jul 26 21:38:34.251329525 (58h3m17.096131051s) [upload=55604]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.33565489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.......X....] duration: 9.45536103s FAIL (2025-07-26 21:40:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.455892373s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)