Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 227.52693ms FAIL (2025-07-28 01:12:11 UTC) orbiter warning: uplink: too many requests Duration: 227.557054ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030815549s FAIL (2025-07-28 01:11:46 UTC) ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== 1:10:46.408263791 (27h19m2.189197174s) [noise=true , address=104.36.82.15:4284 , piece_id=MQ2GUJWUTLDZ75FALBBR3SCOYFZI6DUOC2NTT5D2HDGBUIPFZFAQ, seg_pos={0 0}, upload=26060]: Uploading piece... Jul 28 01:10:46.408865904 (27h19m2.189799291s) [noise=true , address=104.36.82.52:4114 , piece_id=OLDGIRC3436OWCJMEW75NFNU6NF4I2BON6SCLX5B4FVFVYXCU53A, seg_pos={0 0}, upload=26060]: Uploading piece... Jul 28 01:10:46.409283928 (27h19m2.190217308s) [noise=true , address=104.36.82.47:4527 , piece_id=ZYOTYUEIYS6W7ESULWFSQMZJPMHA62WJNCCTSMJYEZ7MVNWOBRUQ, seg_pos={0 0}, upload=26060]: Uploading piece... Jul 28 01:10:46.409668304 (27h19m2.19060169s) [noise=true , address=104.36.82.49:4987 , piece_id=MZT5I7SIVE2H2M2KP7YDATDKTNTSY4T2WA6ORSTERY3MQCTBGKVQ, seg_pos={0 0}, upload=26060]: Uploading piece... Jul 28 01:10:46.410105824 (27h19m2.191039211s) [noise=true , address=104.36.82.53:4127 , piece_id=ATCQDG7V43XZDXTCXLRMFKXQJPWKCARCTSKXF6RZLR6QUMUALD5Q, seg_pos={0 0}, upload=26060]: Uploading piece... Jul 28 01:10:46.411231094 (27h19m2.192164484s) [noise=true , address=104.36.82.41:4794 , piece_id=CGMU2NIHISG2MBRUPRU7XFWHKYA5VBONSP77VYJG7K223V2EXT4Q, seg_pos={0 0}, upload=26060]: Uploading piece... Jul 28 01:10:46.412203723 (27h19m2.193137113s) [noise=true , address=104.36.82.44:4890 , piece_id=7S2CV2AIL2DLVTXNVRQQUNTWFCZMBV4B57G5ZXGZ7J2JPMB4K6YQ, seg_pos={0 0}, upload=26060]: Uploading piece... Jul 28 01:11:46.320354209 (27h20m2.101287601s) [upload=26060]: Next segment err: context deadline exceeded Jul 28 01:11:46.320536285 (27h20m2.101469683s) [noise=true , address=104.36.82.41:4794 , piece_id=CGMU2NIHISG2MBRUPRU7XFWHKYA5VBONSP77VYJG7K223V2EXT4Q, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:12R7HqLyfJRftdvi2pD81NNrRVVnCEyKT9nrzv8eUWU4j2WGmzw, address:104.36.82.41:4794): protocol: write tcp 159.223.143.39:43964->104.36.82.41:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.320686739 (27h20m2.101620117s) [noise=true , address=104.36.82.44:4890 , piece_id=7S2CV2AIL2DLVTXNVRQQUNTWFCZMBV4B57G5ZXGZ7J2JPMB4K6YQ, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: write tcp 159.223.143.39:54790->104.36.82.44:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.320831810 (27h20m2.10176519s) [noise=true , address=104.36.82.42:4806 , piece_id=QKWUHRKE6JLZSREGRQV2PWTQKZRMO3BWHQJVGSLZXLAX25YFIVLQ, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:1tWnxuTSo4yHAJUzvVFTmwPSNb8cBJJJ17BnKd5Z3HMYf66vSC, address:104.36.82.42:4806): protocol: rpc: dial tcp 104.36.82.42:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.320872936 (27h20m2.101806319s) [noise=true , address=104.36.82.47:4527 , piece_id=ZYOTYUEIYS6W7ESULWFSQMZJPMHA62WJNCCTSMJYEZ7MVNWOBRUQ, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: write tcp 159.223.143.39:44416->104.36.82.47:4527: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.320933873 (27h20m2.101867255s) [noise=true , address=104.36.82.50:4043 , piece_id=3PQFO3TBY7KZSKARUDHSQ3QN746XLKFHITN4KN7YVOY5EQRRL2LA, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:104.36.82.50:4043): protocol: write tcp 159.223.143.39:43444->104.36.82.50:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.320965185 (27h20m2.101898565s) [noise=true , address=104.36.82.19:4401 , piece_id=IOW3KDDCHWYLDUHVLF76E35QUVE6WOU5IUM7EPXZFNJDIPCBC2XQ, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:104.36.82.19:4401): protocol: rpc: dial tcp 104.36.82.19:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321032446 (27h20m2.101965828s) [noise=true , address=104.36.82.56:4237 , piece_id=QKYW6ZNI2YEJUCF76WJ6ZU35LFZKPGF2XQI5QHA5U27GEY7UQFBQ, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:1VzewCRbNpJv7kNwp9wruQuoVSaZFjA44qN33Ct7qm1GnfhtDj, address:104.36.82.56:4237): protocol: rpc: dial tcp 104.36.82.56:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321089837 (27h20m2.102023224s) [noise=true , address=104.36.82.15:4284 , piece_id=MQ2GUJWUTLDZ75FALBBR3SCOYFZI6DUOC2NTT5D2HDGBUIPFZFAQ, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:1YAnLu8onh19Mx524VikQzCHEFW4U8dgtuLx8amtQAiReSCisF, address:104.36.82.15:4284): protocol: write tcp 159.223.143.39:53918->104.36.82.15:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321150909 (27h20m2.102084292s) [noise=true , address=104.36.82.8:4979 , piece_id=IXMF7OOVMM3FP4KHGXO7EJVLZJWPV3VYK7VHVTZMD2OV23EWDPYQ, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC, address:104.36.82.8:4979): protocol: write tcp 159.223.143.39:37302->104.36.82.8:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321205183 (27h20m2.102138557s) [noise=true , address=104.36.82.52:4114 , piece_id=OLDGIRC3436OWCJMEW75NFNU6NF4I2BON6SCLX5B4FVFVYXCU53A, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:12PXReFMGqAhCAp5BSE3vsJdfRAUathsJLdB8Guz7DzW6UHmATM, address:104.36.82.52:4114): protocol: write tcp 159.223.143.39:54600->104.36.82.52:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321216770 (27h20m2.10215015s) [noise=true , address=104.36.82.16:4321 , piece_id=LHCPAGKAGF2CHCYCWOAVUGGIW3CAHXRYWEHUL22Q4S6RA5QU4B6A, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae, address:104.36.82.16:4321): protocol: rpc: dial tcp 104.36.82.16:4321: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321284273 (27h20m2.102217654s) [noise=true , address=104.36.82.54:4167 , piece_id=KKPW23SKPB4FERNHI4VJBD27WWT44L7DSNFKB7TSJAXO753NL5XQ, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:104.36.82.54:4167): protocol: write tcp 159.223.143.39:49286->104.36.82.54:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321333110 (27h20m2.10226649s) [noise=true , address=104.36.82.53:4127 , piece_id=ATCQDG7V43XZDXTCXLRMFKXQJPWKCARCTSKXF6RZLR6QUMUALD5Q, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:12VM12t6Dq6SyApviDYSSUqWnnzW3wTfA4o8VsvJaah4fLj9EWh, address:104.36.82.53:4127): protocol: write tcp 159.223.143.39:57054->104.36.82.53:4127: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321344830 (27h20m2.102278192s) [noise=true , address=104.36.82.12:4166 , piece_id=KTLYRFKRYWROK6K7RQRUB5IYDHHTQ4NQXPSULNOA4VA6MNH6F2JA, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321355860 (27h20m2.102289223s) [noise=true , address=104.36.82.49:4987 , piece_id=MZT5I7SIVE2H2M2KP7YDATDKTNTSY4T2WA6ORSTERY3MQCTBGKVQ, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:1oZxiaBQD1grLu8d1HyERweeuXNXc1aBgyAoe1B86gqm6ypvBC, address:104.36.82.49:4987): protocol: write tcp 159.223.143.39:38056->104.36.82.49:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321407900 (27h20m2.102341281s) [noise=true , address=104.36.82.14:4255 , piece_id=DK2KXN4HF6PJHFBNEHCGDKWIJ5IGJ3CNZUF7EWNU6KBAHX7G64KA, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR, address:104.36.82.14:4255): protocol: rpc: dial tcp 104.36.82.14:4255: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321460291 (27h20m2.10239367s) [noise=true , address=104.36.82.55:4194 , piece_id=CKYQOCCDXKKZCFXYO7WVPMILMCUHLQTSYZIEGWFBW4ICYLUEKHFA, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: rpc: dial tcp 104.36.82.55:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321469262 (27h20m2.102402641s) [noise=true , address=104.36.82.21:4493 , piece_id=55IXRYXDIZLHV2Q7HPDLCAXJRY3JEIGF7GVUIOSSMXR435YQL2LQ, seg_pos={0 0}, upload=26060]: Done uploading piece. err: ecclient: upload failed (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD, address:104.36.82.21:4493): protocol: rpc: dial tcp 104.36.82.21:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:46.321513303 (27h20m2.102446681s) [seg_pos={0 0}, upload=26060]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.030837235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX...........X...X...X..................................] duration: 470.042799ms FAIL (2025-07-28 01:11:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.07829ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 202.558385ms FAIL (2025-07-28 01:11:10 UTC) orbiter warning: uplink: too many requests Duration: 202.591627ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X...................X....XX........X..X...........] duration: 473.317522ms FAIL (2025-07-28 01:11:03 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 473.351436ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031984591s FAIL (2025-07-28 01:10:46 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== oading piece... Jul 28 01:09:46.309964751 (27h18m2.090898115s) [noise=true , address=104.36.82.52:4108 , piece_id=OQZ6WPXNFAAL4XGXPCD43GX7XFQXXHYDNMIYLO3YDF73C4UD5IAA, seg_pos={0 0}, upload=26046]: Uploading piece... Jul 28 01:09:46.310643206 (27h18m2.091576596s) [noise=true , address=104.36.82.12:4173 , piece_id=ZC3D2HK5JIZFBBMDHEQIXPE3VPCGNPULANMDL6QXDZ2RKTKWO4WQ, seg_pos={0 0}, upload=26046]: Uploading piece... Jul 28 01:09:46.316363407 (27h18m2.097296795s) [noise=true , address=104.36.82.53:4149 , piece_id=CNVFJCXKQB3S73SIRD6DFWKB5D6Y4T5U3VJFOWTAJRL2UTU33CJQ, seg_pos={0 0}, upload=26046]: Uploading piece... Jul 28 01:09:46.316969143 (27h18m2.097902555s) [noise=true , address=104.36.82.41:4766 , piece_id=UAQHTFDAVUJFXGYRMTKOXJ5EUO73XYATV4PDLYJ66VX3WRU3NCAA, seg_pos={0 0}, upload=26046]: Uploading piece... Jul 28 01:09:46.317587000 (27h18m2.098520393s) [noise=true , address=104.36.82.46:4506 , piece_id=57EA2MJTVPUZ5RDKP4JNRXFU7ZWHMUBMDVUXW7O6MOXISOQKYHOA, seg_pos={0 0}, upload=26046]: Uploading piece... Jul 28 01:09:46.317996032 (27h18m2.098929412s) [noise=true , address=104.36.82.5:4861 , piece_id=H2R2V6U7BWKH3K4LD6IXSNUYKQXAQZTVPLML6KIZC36A6DPKIFKQ, seg_pos={0 0}, upload=26046]: Uploading piece... Jul 28 01:09:46.318374418 (27h18m2.099307806s) [noise=true , address=104.36.82.6:4879 , piece_id=5WT43DCMP2NR4RSC6E7IPFEPNREB6SV2MC6AOUHZYEI3HDA6ODMA, seg_pos={0 0}, upload=26046]: Uploading piece... Jul 28 01:09:46.318744261 (27h18m2.099677648s) [noise=true , address=64.49.13.169:4709 , piece_id=54DVUM6S3U6WPVKTTJ3XET5C6KCM465Z27J55CT3LXUV4Z3R7WIA, seg_pos={0 0}, upload=26046]: Uploading piece... Jul 28 01:09:46.319174112 (27h18m2.100107501s) [noise=true , address=104.36.82.56:4264 , piece_id=3A5M3ZPE4WY744KPJA2Y2JWG27XO4INH7QPRJU7GXZUHCDTGGZWA, seg_pos={0 0}, upload=26046]: Uploading piece... Jul 28 01:09:46.424232881 (27h18m2.205166264s) [upload=26046]: Next returned nil segment Jul 28 01:09:46.424737029 (27h18m2.205670418s) [upload=26046]: Waiting for error group managing segments... Jul 28 01:09:46.489830036 (27h18m2.270763407s) [noise=true , address=77.90.31.10:4550 , piece_id=JX3XYJA7E5ZKPTULFOSTXGVMO5BKLNJFDBHMSLVD7DGF57YIGVVQ, seg_pos={0 0}, upload=26046]: Done uploading piece. err: <nil> Jul 28 01:09:46.975840957 (27h18m2.756774341s) [noise=true , address=64.49.13.169:4709 , piece_id=54DVUM6S3U6WPVKTTJ3XET5C6KCM465Z27J55CT3LXUV4Z3R7WIA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: <nil> Jul 28 01:10:46.222320267 (27h19m2.003253643s) [noise=true , address=104.36.82.15:4276 , piece_id=ZE2XUKJWO7QPGJFIMALRHATJMCY6V2O4ORVJPYNRBOQLYZVXRIZA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.222410359 (27h19m2.003343737s) [noise=true , address=104.36.82.49:4985 , piece_id=HMSUZTBNC6X23CUZOC4CHKIQZ7FUURCUOQALD7KZQRR5HQGA27EA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: rpc: dial tcp 104.36.82.49:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.222482306 (27h19m2.003415685s) [noise=true , address=104.36.82.20:4445 , piece_id=NUEAQEBHWEFA5QETI2WFJNFDQPCH5IZKO5LZS7LCC2DZGD6L3HAA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:1zCSLthypJ1VkjhPxE5bTo35d9kUgQch1Wbj8MU7tK4khzCbcN, address:104.36.82.20:4445): protocol: rpc: dial tcp 104.36.82.20:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.222550754 (27h19m2.003484131s) [noise=true , address=104.36.82.6:4879 , piece_id=5WT43DCMP2NR4RSC6E7IPFEPNREB6SV2MC6AOUHZYEI3HDA6ODMA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:12jH7W99nQ77T67b8WVVqmg9Jqi5QoB1qM5Wcj8WexywAZ7fRNT, address:104.36.82.6:4879): protocol: rpc: dial tcp 104.36.82.6:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.222621012 (27h19m2.003554389s) [noise=true , address=104.36.82.55:4215 , piece_id=HCBL34TD242GFSMUYX2O57X473GF32HXOOHZNCURM6CAXCT4NYVQ, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223274399 (27h19m2.004207789s) [noise=true , address=104.36.82.54:4189 , piece_id=KQYGYLCP6BMP64ZVP3GWY423MCNPPLXOSVAZYQ637MCUY67WTASA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:104.36.82.54:4189): protocol: rpc: dial tcp 104.36.82.54:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223360431 (27h19m2.004293811s) [noise=true , address=104.36.82.53:4149 , piece_id=CNVFJCXKQB3S73SIRD6DFWKB5D6Y4T5U3VJFOWTAJRL2UTU33CJQ, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: rpc: dial tcp 104.36.82.53:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223431817 (27h19m2.004365195s) [noise=true , address=104.36.82.46:4506 , piece_id=57EA2MJTVPUZ5RDKP4JNRXFU7ZWHMUBMDVUXW7O6MOXISOQKYHOA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: rpc: dial tcp 104.36.82.46:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223506151 (27h19m2.004439529s) [noise=true , address=104.36.82.5:4861 , piece_id=H2R2V6U7BWKH3K4LD6IXSNUYKQXAQZTVPLML6KIZC36A6DPKIFKQ, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: rpc: dial tcp 104.36.82.5:4861: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223580456 (27h19m2.004513834s) [noise=true , address=104.36.82.56:4264 , piece_id=3A5M3ZPE4WY744KPJA2Y2JWG27XO4INH7QPRJU7GXZUHCDTGGZWA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: write tcp 159.223.143.39:48012->104.36.82.56:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223739144 (27h19m2.004672521s) [noise=true , address=104.36.82.14:4242 , piece_id=VATIX5ZAPVNQRASYTELPLY7UXE6BQYRAI5GU4X2QR46LSWRV2HCA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz, address:104.36.82.14:4242): protocol: rpc: dial tcp 104.36.82.14:4242: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223830923 (27h19m2.004764304s) [noise=true , address=104.36.82.21:4473 , piece_id=2TMALR2KPWHXO755X5FOLQJQX3K7OF3NYLXRLQL3QU32TLBEJCEA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:104.36.82.21:4473): protocol: rpc: dial tcp 104.36.82.21:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223865123 (27h19m2.004798501s) [noise=true , address=104.36.82.41:4766 , piece_id=UAQHTFDAVUJFXGYRMTKOXJ5EUO73XYATV4PDLYJ66VX3WRU3NCAA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: rpc: dial tcp 104.36.82.41:4766: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223906183 (27h19m2.004839564s) [noise=true , address=104.36.82.47:4522 , piece_id=G5LYEQQEEGNTTTOJECHOEML6HCL33FO64HJLSDNHH3CVXLL3HUYA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: rpc: dial tcp 104.36.82.47:4522: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223949856 (27h19m2.004883234s) [noise=true , address=104.36.82.52:4108 , piece_id=OQZ6WPXNFAAL4XGXPCD43GX7XFQXXHYDNMIYLO3YDF73C4UD5IAA, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:12XJJZCUSHUS52YC6jhoPB4ayjsyKZT477PymPj4inkZtdUvheT, address:104.36.82.52:4108): protocol: rpc: dial tcp 104.36.82.52:4108: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223974768 (27h19m2.004908149s) [noise=true , address=104.36.82.12:4173 , piece_id=ZC3D2HK5JIZFBBMDHEQIXPE3VPCGNPULANMDL6QXDZ2RKTKWO4WQ, seg_pos={0 0}, upload=26046]: Done uploading piece. err: ecclient: upload failed (node:12HbGUhwH2eNqY75sLQCDgkvJ1QSrJ2n4Qg8A5Z73dFXxRpVwSC, address:104.36.82.12:4173): protocol: write tcp 159.223.143.39:35532->104.36.82.12:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:46.223998056 (27h19m2.004931424s) [seg_pos={0 0}, upload=26046]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 01:10:46.251599451 (27h19m2.032532828s) [upload=26046]: Done uploading segments... Jul 28 01:10:46.251637451 (27h19m2.03257082s) [upload=26046]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.032015205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX...........X...X...X.................................] duration: 630.774981ms FAIL (2025-07-28 01:10:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 630.807935ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 275.83153ms FAIL (2025-07-28 01:10:07 UTC) orbiter warning: uplink: too many requests Duration: 275.867355ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X...................X....XX........X..X..........] duration: 976.608086ms FAIL (2025-07-28 01:10:00 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 976.635173ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XX...........X...X...X................................] duration: 450.262039ms FAIL (2025-07-28 01:09:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.298078ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029923347s FAIL (2025-07-28 01:09:46 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== }, upload=26031]: Uploading piece... Jul 28 01:08:46.218642823 (27h17m1.99957621s) [noise=true , address=104.36.82.53:4130 , piece_id=4ZJ7OPFTOBC322S446ZDZ4VV3HG5GOKAPMTMPJ7VUHVIHGCTKMEQ, seg_pos={0 0}, upload=26031]: Uploading piece... Jul 28 01:08:46.219164078 (27h17m2.000097709s) [noise=true , address=104.36.82.21:4508 , piece_id=XU3S3V2YRQCZGJJKI4S4VMYAWT5UJ7UN6HVFRINA6J6XECI56YSA, seg_pos={0 0}, upload=26031]: Uploading piece... Jul 28 01:08:46.219576904 (27h17m2.00051029s) [noise=true , address=104.36.82.41:4782 , piece_id=SXQX3G2HHD6BBJ4M3U5ASXIJ3ZY7YTE3RAHGZJCHLIQ542JWHYQA, seg_pos={0 0}, upload=26031]: Uploading piece... Jul 28 01:08:46.219958052 (27h17m2.000891439s) [noise=true , address=104.36.82.44:4898 , piece_id=OYNVXW55HGK5QVLSMWAQHBSICNV2ELK7DXRGKJG572PBZCJVRFJA, seg_pos={0 0}, upload=26031]: Uploading piece... Jul 28 01:08:46.220420943 (27h17m2.001354332s) [noise=true , address=104.36.82.42:4808 , piece_id=BMUZXV4KU5EANS5KUCMMBA4K5JVG5A7LNORYASWTSHS7O5EUEP4Q, seg_pos={0 0}, upload=26031]: Uploading piece... Jul 28 01:08:46.220800772 (27h17m2.001734155s) [noise=true , address=104.36.82.55:4225 , piece_id=HWRRGFIMSGD7UIR6L45NJQXWMSB7ZZ7AN772Q3A7CXE6D4CJ7IUA, seg_pos={0 0}, upload=26031]: Uploading piece... Jul 28 01:08:46.221288125 (27h17m2.002221512s) [noise=true , address=104.36.82.10:4090 , piece_id=K6QO3OVWD3EZ2SR3463TWRFNGCT6S7WA7YYDXCUZMEHEGMYU6HHA, seg_pos={0 0}, upload=26031]: Uploading piece... Jul 28 01:08:46.221735889 (27h17m2.002669272s) [noise=true , address=104.36.82.11:4124 , piece_id=4SCL5C3BHAMXWJ4AU6U5XAII55AJEE26S3XGHU3WWONWXCIA4TJA, seg_pos={0 0}, upload=26031]: Uploading piece... Jul 28 01:08:46.441047130 (27h17m2.221980546s) [upload=26031]: Next returned nil segment Jul 28 01:08:46.441066836 (27h17m2.222000211s) [upload=26031]: Waiting for error group managing segments... Jul 28 01:08:48.755221287 (27h17m4.536154656s) [noise=true , address=79.127.201.215:4347 , piece_id=UULSST4QBTDHIDTQ4TRLLDLKJCGCK5TJCEY7X6ODMYYNS3LST74Q, seg_pos={0 0}, upload=26031]: Done uploading piece. err: <nil> Jul 28 01:09:46.135532857 (27h18m1.916466239s) [noise=true , address=104.36.82.44:4898 , piece_id=OYNVXW55HGK5QVLSMWAQHBSICNV2ELK7DXRGKJG572PBZCJVRFJA, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.135638925 (27h18m1.91657231s) [noise=true , address=104.36.82.13:4205 , piece_id=V7SVHLPVHCF3FBKUZ72R3MI3KFCBOUYUINSM3E3MBW3WIXSC4XDA, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr, address:104.36.82.13:4205): protocol: rpc: dial tcp 104.36.82.13:4205: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.135705816 (27h18m1.916639194s) [noise=true , address=104.36.82.53:4130 , piece_id=4ZJ7OPFTOBC322S446ZDZ4VV3HG5GOKAPMTMPJ7VUHVIHGCTKMEQ, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:12N8MGboFyEV381rkrBva3G3advz3QQEY2HCHxwqzKX2X5zxNvb, address:104.36.82.53:4130): protocol: rpc: dial tcp 104.36.82.53:4130: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.135773114 (27h18m1.916706492s) [noise=true , address=104.36.82.6:4888 , piece_id=3KMFDRMZSVL23F3ZR7ICIQ6YRXKEXQVSYNCRJTWCGCHEPE52EBCQ, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:12HwYszCTcXEHqSNnWJpT11NpKg8cFUvdAqU2wUigsLbxmDFkg6, address:104.36.82.6:4888): protocol: rpc: dial tcp 104.36.82.6:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.135843526 (27h18m1.916776904s) [noise=true , address=104.36.82.45:4915 , piece_id=5L4PNPZKOQDCELI24ZQQVHJUTMQOQKQCG7OAGE7THRIWWMHJ5L3Q, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT, address:104.36.82.45:4915): protocol: rpc: dial tcp 104.36.82.45:4915: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.135926208 (27h18m1.916859585s) [noise=true , address=104.36.82.21:4508 , piece_id=XU3S3V2YRQCZGJJKI4S4VMYAWT5UJ7UN6HVFRINA6J6XECI56YSA, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3, address:104.36.82.21:4508): protocol: rpc: dial tcp 104.36.82.21:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.135988582 (27h18m1.916921964s) [noise=true , address=104.36.82.11:4124 , piece_id=4SCL5C3BHAMXWJ4AU6U5XAII55AJEE26S3XGHU3WWONWXCIA4TJA, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:1QAgfinEkxYwjzvtr53FRritkoDNbfYAw21qcJuj3uCY4JT4cZ, address:104.36.82.11:4124): protocol: rpc: dial tcp 104.36.82.11:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.136059645 (27h18m1.916993024s) [noise=true , address=104.36.82.40:4962 , piece_id=5ZFTOLZJQUB6APOXWDT3EJKSZTRCRJZMB5DTMG5GPZSGZOW42CQA, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3, address:104.36.82.40:4962): protocol: rpc: dial tcp 104.36.82.40:4962: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.136123699 (27h18m1.917057076s) [noise=true , address=104.36.82.19:4427 , piece_id=TIV225G64DTVGWIDYPJLWXIPFIDPEDVT6IXGLUKDZ3NALCU5SM7Q, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.136373727 (27h18m1.917307104s) [noise=true , address=104.36.82.12:4164 , piece_id=BUZZS24RYMNPI4PSQNOQXIXQJYBWM2R3U4EDWXUQUGJBEGWFHZ7A, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:12iQgyDiz9vu1TbCRuATGmnrfbgBaia72b77ZjQJ6gpnEFbQzTJ, address:104.36.82.12:4164): protocol: rpc: dial tcp 104.36.82.12:4164: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.136478817 (27h18m1.917412201s) [noise=true , address=104.36.82.7:4927 , piece_id=PO7A6JZCUMBIFHBS3MZMKF77IHKBFIHFKDM4B6SUHEOQKNW7ZAGQ, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:1AD5Ju87nY6LL5aTbFFsfNJRt8gkyGExsSrJLf6AP2q1nd9HBr, address:104.36.82.7:4927): protocol: rpc: dial tcp 104.36.82.7:4927: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.136545557 (27h18m1.917478942s) [noise=true , address=104.36.82.54:4188 , piece_id=BFF5DMFTOCLMF3JDAWI3NWK722XR2576HGZJBQGZPTRKHYQYZX5Q, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG, address:104.36.82.54:4188): protocol: rpc: dial tcp 104.36.82.54:4188: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.136612029 (27h18m1.917545412s) [noise=true , address=104.36.82.55:4225 , piece_id=HWRRGFIMSGD7UIR6L45NJQXWMSB7ZZ7AN772Q3A7CXE6D4CJ7IUA, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: rpc: dial tcp 104.36.82.55:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.136676950 (27h18m1.917610334s) [noise=true , address=104.36.82.10:4090 , piece_id=K6QO3OVWD3EZ2SR3463TWRFNGCT6S7WA7YYDXCUZMEHEGMYU6HHA, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:104.36.82.10:4090): protocol: rpc: dial tcp 104.36.82.10:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.136744558 (27h18m1.917677943s) [noise=true , address=104.36.82.52:4120 , piece_id=J7RD7GHYCKEBSI2JWBN3HYND347XYNJMDI6X7D3KH7WV35QU42DA, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: rpc: dial tcp 104.36.82.52:4120: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.136811895 (27h18m1.917745278s) [noise=true , address=104.36.82.41:4782 , piece_id=SXQX3G2HHD6BBJ4M3U5ASXIJ3ZY7YTE3RAHGZJCHLIQ542JWHYQA, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:1vst2WW2D3NGbxPDDkjFQPxuwX3joAY77EdWCGXxU3H1NVGgcr, address:104.36.82.41:4782): protocol: rpc: dial tcp 104.36.82.41:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.136875948 (27h18m1.917809333s) [noise=true , address=104.36.82.42:4808 , piece_id=BMUZXV4KU5EANS5KUCMMBA4K5JVG5A7LNORYASWTSHS7O5EUEP4Q, seg_pos={0 0}, upload=26031]: Done uploading piece. err: ecclient: upload failed (node:1dBGnojgbYNwFfATQTahFBnkvZCGyZLiwRBa3AYv4iVY4U3yE2, address:104.36.82.42:4808): protocol: write tcp 159.223.143.39:39310->104.36.82.42:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:46.136899255 (27h18m1.917832648s) [seg_pos={0 0}, upload=26031]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:09:46.161292793 (27h18m1.942226171s) [upload=26031]: Done uploading segments... Jul 28 01:09:46.161324477 (27h18m1.942257859s) [upload=26031]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.029954857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 214.769248ms FAIL (2025-07-28 01:09:46 UTC) orbiter warning: uplink: too many requests Duration: 214.808143ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 196.495763ms FAIL (2025-07-28 01:08:55 UTC) orbiter warning: uplink: too many requests Duration: 196.532595ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...........X...................X....XX........X..X.........] duration: 414.284795ms FAIL (2025-07-28 01:08:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 414.315932ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XX...........X...X...X...............................] duration: 457.053381ms FAIL (2025-07-28 01:08:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 457.090517ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.041981678s FAIL (2025-07-28 01:08:43 UTC) ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== Z67UARQ54WIBBQBYKZVGK2CYRA, seg_pos={0 0}, upload=26016]: Uploading piece... Jul 28 01:07:43.904436806 (27h15m59.685370194s) [noise=true , address=104.36.82.53:4122 , piece_id=QJGZG4XKDSUJEMXHSP6GNHZ7VAMIUJ4UKR7Q3ZPIA6ZOZNKAKN3Q, seg_pos={0 0}, upload=26016]: Uploading piece... Jul 28 01:07:43.904853030 (27h15m59.685786418s) [noise=true , address=104.36.82.54:4175 , piece_id=HGDJ2IPBYNA2LM4IGIKW3CV36GXVIDFPS74KZHMHGWXKROCQOQYA, seg_pos={0 0}, upload=26016]: Uploading piece... Jul 28 01:07:43.905214550 (27h15m59.686147935s) [noise=true , address=104.36.82.40:4958 , piece_id=MYC2EQAMQBXHTBEV6PHUHX7B42OUUNMP24N7ZBH4D54W7SP4IQVA, seg_pos={0 0}, upload=26016]: Uploading piece... Jul 28 01:07:43.905568305 (27h15m59.686501681s) [noise=true , address=104.36.82.2:4764 , piece_id=WVTZUSSNX2OOWGPISD3GXIVOKKIXCQIISQEDIYBSKWOWKY4XC4PA, seg_pos={0 0}, upload=26016]: Uploading piece... Jul 28 01:07:43.905982470 (27h15m59.686915855s) [noise=true , address=104.36.82.8:4982 , piece_id=EDJ3OX6GLG5ORKYJLSXVTR7W2UA6NOQLYAJ3DFAFHCAZE2GECYKQ, seg_pos={0 0}, upload=26016]: Uploading piece... Jul 28 01:07:43.906350143 (27h15m59.687283529s) [noise=true , address=104.36.82.14:4225 , piece_id=Q56IC3VGUCBJWBDYSRDRGDQGYX7HBJSS4SOG4R2LLMSVQERT4SIA, seg_pos={0 0}, upload=26016]: Uploading piece... Jul 28 01:07:43.906729029 (27h15m59.687662412s) [noise=true , address=104.36.82.5:4861 , piece_id=43IAGYB475VO5NYXTD3KQICTY5YVQ3DCEQ3QAHCLVBYWHPJH6S6Q, seg_pos={0 0}, upload=26016]: Uploading piece... Jul 28 01:07:43.907167015 (27h15m59.688100399s) [noise=true , address=104.36.82.42:4805 , piece_id=3AVJDJIV35ND6YQGTZNNCXQZIFXWIRK5RLVYOQFLZ54MMRRLJDMQ, seg_pos={0 0}, upload=26016]: Uploading piece... Jul 28 01:08:43.822841460 (27h16m59.603774847s) [upload=26016]: Next segment err: context deadline exceeded Jul 28 01:08:43.824548059 (27h16m59.605481443s) [noise=true , address=104.36.82.13:4199 , piece_id=ODJ7I4TRC33BQXIGAIE4LV5575Z67UARQ54WIBBQBYKZVGK2CYRA, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:1DagQMPpe9F6w5xewXXCfzVBPcmgg6EKCauFsNCWHoHdP56bs7, address:104.36.82.13:4199): protocol: rpc: dial tcp 104.36.82.13:4199: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.824629938 (27h16m59.605563315s) [noise=true , address=104.36.82.11:4119 , piece_id=MJWBQC7GVFWUAKWIS6VCDONDIAVHG7YVLCCLQFKEBO42F7W7JG7A, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:104.36.82.11:4119): protocol: rpc: dial tcp 104.36.82.11:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.824701907 (27h16m59.605635286s) [noise=true , address=104.36.82.54:4175 , piece_id=HGDJ2IPBYNA2LM4IGIKW3CV36GXVIDFPS74KZHMHGWXKROCQOQYA, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.824759383 (27h16m59.60569276s) [noise=true , address=104.36.82.42:4805 , piece_id=3AVJDJIV35ND6YQGTZNNCXQZIFXWIRK5RLVYOQFLZ54MMRRLJDMQ, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:1nh6Kih617STw2M8iHVjs4xP1HjjckJHyUxQ5z6ZFRhWwG6RXi, address:104.36.82.42:4805): protocol: rpc: dial tcp 104.36.82.42:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.824870585 (27h16m59.605803963s) [noise=true , address=104.36.82.53:4122 , piece_id=QJGZG4XKDSUJEMXHSP6GNHZ7VAMIUJ4UKR7Q3ZPIA6ZOZNKAKN3Q, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: rpc: dial tcp 104.36.82.53:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.824911055 (27h16m59.605844441s) [noise=true , address=104.36.82.55:4215 , piece_id=IJGV4YHV7OLRIV4BGHLDGSSTDPJORJBBAGSGWHALKXJWO4FUFGKQ, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.824956555 (27h16m59.60588993s) [noise=true , address=104.36.82.40:4958 , piece_id=MYC2EQAMQBXHTBEV6PHUHX7B42OUUNMP24N7ZBH4D54W7SP4IQVA, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:1ySYdoPKsX8RRcnfGuneBqVjXLbuDQJ7ofRiv6dEyYxR32hoGR, address:104.36.82.40:4958): protocol: rpc: dial tcp 104.36.82.40:4958: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.824989757 (27h16m59.605923134s) [noise=true , address=104.36.82.3:4789 , piece_id=4ODORPDXH4ZJBT547MJKSCQYFBWUVK7XNTUMGAM27TZHYZ5DYMHA, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:12eQnnZm3aoAEA6cHJshERFmZuyh5cUjqzYK687CS27CQ8nGzrT, address:104.36.82.3:4789): protocol: rpc: dial tcp 104.36.82.3:4789: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.825025240 (27h16m59.60595862s) [noise=true , address=104.36.82.2:4764 , piece_id=WVTZUSSNX2OOWGPISD3GXIVOKKIXCQIISQEDIYBSKWOWKY4XC4PA, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:12hC8kZEorpoPUHZE9fKmpYEEG6HrSWPkn574p4LehpoGwPHPdt, address:104.36.82.2:4764): protocol: rpc: dial tcp 104.36.82.2:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.825059943 (27h16m59.60599333s) [noise=true , address=104.36.82.47:4538 , piece_id=DZ5ONMNLBWRSJR6YETEAKQEIJU45ED7VXUQQ6NPA3KBW7HNLVEGQ, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.825103248 (27h16m59.606036629s) [noise=true , address=104.36.82.14:4225 , piece_id=Q56IC3VGUCBJWBDYSRDRGDQGYX7HBJSS4SOG4R2LLMSVQERT4SIA, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:104.36.82.14:4225): protocol: rpc: dial tcp 104.36.82.14:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.825150255 (27h16m59.606083632s) [noise=true , address=104.36.82.56:4264 , piece_id=UGFSJQQ22INVOQ3SISATTSGO5PM6LQZCG3LEP6TU2VK7IT6XFD6A, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: write tcp 159.223.143.39:46722->104.36.82.56:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.825205709 (27h16m59.606139086s) [noise=true , address=104.36.82.5:4861 , piece_id=43IAGYB475VO5NYXTD3KQICTY5YVQ3DCEQ3QAHCLVBYWHPJH6S6Q, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: rpc: dial tcp 104.36.82.5:4861: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.825244912 (27h16m59.606178291s) [noise=true , address=104.36.82.8:4982 , piece_id=EDJ3OX6GLG5ORKYJLSXVTR7W2UA6NOQLYAJ3DFAFHCAZE2GECYKQ, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:17EBpouVrJJDho24tucF5w43aFMqjeMviU3VoRCoL2b2Ho8WNU, address:104.36.82.8:4982): protocol: rpc: dial tcp 104.36.82.8:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.825280701 (27h16m59.606214078s) [noise=true , address=104.36.82.50:4026 , piece_id=S6KVHIURAG4LA42XLRJ4BWYDCKHKYJA74OBRFSCEAAHEYOKMS3XQ, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:12o2Lo1F6o5UE1rDmuoW27tACyupSwFb9GEVfFNcLTGEAguVPwG, address:104.36.82.50:4026): protocol: rpc: dial tcp 104.36.82.50:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.825323168 (27h16m59.606256544s) [noise=true , address=104.36.82.7:4931 , piece_id=CX3CKJWS4N5OJNQWH4P7X5QRIVO5OVXXE4XU4T2TH6IGMUNXOTKA, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:125rm4HueKsNdwL3eCTCSWoPseR2LQWcsE3jNPpxSfYQQorj9LL, address:104.36.82.7:4931): protocol: rpc: dial tcp 104.36.82.7:4931: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.825342149 (27h16m59.606275524s) [noise=true , address=104.36.82.15:4263 , piece_id=HE4VFVMUUXNWOCDODOLQY2S5LTV4CEHXCAGC7UYY67QB2PXDKQSA, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: rpc: dial tcp 104.36.82.15:4263: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.825385346 (27h16m59.606318716s) [noise=true , address=104.36.82.21:4475 , piece_id=DNKUE3ZYAVVQEJBZWLLJMEQQWR24IOA2RRM4XAY3KZQWZPOZFTMA, seg_pos={0 0}, upload=26016]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:104.36.82.21:4475): protocol: write tcp 159.223.143.39:50678->104.36.82.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:43.825453756 (27h16m59.606387126s) [seg_pos={0 0}, upload=26016]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.042005331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 232.795034ms FAIL (2025-07-28 01:08:38 UTC) orbiter warning: uplink: too many requests Duration: 232.829124ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...........X...................X....XX........X..X........] duration: 426.266635ms FAIL (2025-07-28 01:08:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 426.295458ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 197.969907ms FAIL (2025-07-28 01:08:15 UTC) orbiter warning: uplink: too many requests Duration: 198.00045ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........XX...........X...X...X..............................] duration: 564.02013ms FAIL (2025-07-28 01:08:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 564.055752ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.040149178s FAIL (2025-07-28 01:07:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ce... Jul 28 01:06:43.780963271 (27h14m59.561896647s) [noise=true , address=104.36.82.15:4288 , piece_id=MKB6ENQ2GPLJ7Q44NHPI3W4CHVFDJFU6PWET4TTZO6BCXDZALXXQ, seg_pos={0 0}, upload=25999]: Uploading piece... Jul 28 01:06:43.781235487 (27h14m59.562168869s) [noise=true , address=104.36.82.14:4239 , piece_id=6CGLAIPEAURIYQTLVKVMRENWTRCMCBK3GXY5SDKGFRWLPHT7JRFQ, seg_pos={0 0}, upload=25999]: Uploading piece... Jul 28 01:06:43.781624344 (27h14m59.56255773s) [noise=true , address=104.36.82.10:4110 , piece_id=KHYZJHL3JJC37GWVIN2W4JD3IIHQ56UNLFJC2QDANEPIVGNYQT6A, seg_pos={0 0}, upload=25999]: Uploading piece... Jul 28 01:06:43.782038528 (27h14m59.562971916s) [noise=true , address=104.36.82.53:4123 , piece_id=ZBJDH77OLDKTKZB6LPSONXO3OO2LGWIFDTC3HSZOP2ARO5YYVGCA, seg_pos={0 0}, upload=25999]: Uploading piece... Jul 28 01:06:43.782421309 (27h14m59.563354696s) [noise=true , address=104.36.82.44:4887 , piece_id=GGHL3GOUP6UIJGTBI3EZD3ZJNQZJ4UC32ITARDSVYKQBZZG7EC6A, seg_pos={0 0}, upload=25999]: Uploading piece... Jul 28 01:06:43.782794905 (27h14m59.563728297s) [noise=true , address=104.36.82.42:4818 , piece_id=LVTK2UMX3XAKKGWOM3CVRXNHIGFVUZ5VEFG2FDFUXISC4YJTU3OA, seg_pos={0 0}, upload=25999]: Uploading piece... Jul 28 01:06:43.783208801 (27h14m59.564142185s) [noise=true , address=104.36.82.51:4074 , piece_id=ZH36KLCLSF4YRH7JWHQ754335WPMJDODA7STOULQCB2L6TW66GGA, seg_pos={0 0}, upload=25999]: Uploading piece... Jul 28 01:06:43.783531909 (27h14m59.564465284s) [noise=true , address=104.36.82.13:4212 , piece_id=QID4FI7BDOKSWIRJGLLRVXGVE4CCD77UJDKHB4ADCUOC273ELT3Q, seg_pos={0 0}, upload=25999]: Uploading piece... Jul 28 01:06:43.783830289 (27h14m59.564763673s) [noise=true , address=64.49.13.138:4482 , piece_id=GZQQOX4SL3XCRMXNGSEP74YY6DUOT4UC3D7FFI3GS2KRAK7KHYXA, seg_pos={0 0}, upload=25999]: Uploading piece... Jul 28 01:06:44.255757479 (27h15m0.036690856s) [upload=25999]: Next returned nil segment Jul 28 01:06:44.255777802 (27h15m0.036711184s) [upload=25999]: Waiting for error group managing segments... Jul 28 01:06:44.372729689 (27h15m0.153663059s) [noise=true , address=79.127.205.233:4641 , piece_id=MSTFUYKWNQW7Q5EYRS5YEHKYKRJ7YV4X6YI7OLCURPEJBF4WC7QA, seg_pos={0 0}, upload=25999]: Done uploading piece. err: <nil> Jul 28 01:06:44.610409794 (27h15m0.391343159s) [noise=true , address=64.49.13.138:4482 , piece_id=GZQQOX4SL3XCRMXNGSEP74YY6DUOT4UC3D7FFI3GS2KRAK7KHYXA, seg_pos={0 0}, upload=25999]: Done uploading piece. err: <nil> Jul 28 01:07:43.703190195 (27h15m59.48412357s) [noise=true , address=104.36.82.49:4004 , piece_id=IJTIYSWW6X4FUNK7RXYJXYWE2EIFVEUIQXVMZZ7SNA5TXASAPIEQ, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:1iSAPoDgvJmcbXpHKpVkpNHzso3rvFX5PJa8PooBmohS3BgGKZ, address:104.36.82.49:4004): protocol: rpc: dial tcp 104.36.82.49:4004: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703271635 (27h15m59.484205006s) [noise=true , address=104.36.82.15:4288 , piece_id=MKB6ENQ2GPLJ7Q44NHPI3W4CHVFDJFU6PWET4TTZO6BCXDZALXXQ, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:1gPjPGuueZnizLT2QvJN9y7riwKiW5TWMkQtmGGPP7VZvkNoB7, address:104.36.82.15:4288): protocol: rpc: dial tcp 104.36.82.15:4288: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703347949 (27h15m59.484281315s) [noise=true , address=104.36.82.14:4239 , piece_id=6CGLAIPEAURIYQTLVKVMRENWTRCMCBK3GXY5SDKGFRWLPHT7JRFQ, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:104.36.82.14:4239): protocol: rpc: dial tcp 104.36.82.14:4239: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703402390 (27h15m59.484335765s) [noise=true , address=104.36.82.54:4168 , piece_id=TJLMKGYPKFTNNHERVT5FGC37NJRN347LPQQYOJ4H5H5GZDWNP7SQ, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: rpc: dial tcp 104.36.82.54:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703458772 (27h15m59.484392137s) [noise=true , address=104.36.82.42:4818 , piece_id=LVTK2UMX3XAKKGWOM3CVRXNHIGFVUZ5VEFG2FDFUXISC4YJTU3OA, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:104.36.82.42:4818): protocol: write tcp 159.223.143.39:40970->104.36.82.42:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703507417 (27h15m59.484440782s) [noise=true , address=104.36.82.52:4101 , piece_id=KWF7RCRVL3AS2BI5L3GJPMQQVKJRFPV3YKTCGTE7ADXGXJJ7CN7A, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703569131 (27h15m59.484502508s) [noise=true , address=104.36.82.10:4110 , piece_id=KHYZJHL3JJC37GWVIN2W4JD3IIHQ56UNLFJC2QDANEPIVGNYQT6A, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi, address:104.36.82.10:4110): protocol: write tcp 159.223.143.39:38866->104.36.82.10:4110: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703580100 (27h15m59.484513469s) [noise=true , address=104.36.82.53:4123 , piece_id=ZBJDH77OLDKTKZB6LPSONXO3OO2LGWIFDTC3HSZOP2ARO5YYVGCA, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:104.36.82.53:4123): protocol: rpc: dial tcp 104.36.82.53:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703644956 (27h15m59.484578331s) [noise=true , address=104.36.82.5:4842 , piece_id=3OXELWOI7CTUBOBRRC4CL3TJBPHP3K32AFJ4PSXO4FJ322JKUXOQ, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:07:43.703661958 (27h15m59.484595333s) [noise=true , address=104.36.82.51:4074 , piece_id=ZH36KLCLSF4YRH7JWHQ754335WPMJDODA7STOULQCB2L6TW66GGA, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc, address:104.36.82.51:4074): protocol: rpc: dial tcp 104.36.82.51:4074: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703709881 (27h15m59.484643246s) [noise=true , address=104.36.82.44:4887 , piece_id=GGHL3GOUP6UIJGTBI3EZD3ZJNQZJ4UC32ITARDSVYKQBZZG7EC6A, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H, address:104.36.82.44:4887): protocol: rpc: dial tcp 104.36.82.44:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703741125 (27h15m59.4846745s) [noise=true , address=104.36.82.13:4212 , piece_id=QID4FI7BDOKSWIRJGLLRVXGVE4CCD77UJDKHB4ADCUOC273ELT3Q, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:1itAUtWeFERY4AKhPxE2CRwymp8qNC6wu6uhPt2n6aK58LRu24, address:104.36.82.13:4212): protocol: rpc: dial tcp 104.36.82.13:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703782776 (27h15m59.484716148s) [noise=true , address=104.36.82.12:4179 , piece_id=67T2MG5HVSZPTXGNIWS4CTOPWV643K2GYDB6XHPZWIBMEVLKMR2A, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:129pP9f3rFfTvP6MhG894N7WHuJRZP8aWhzJf1zBwZ76wbzVbZa, address:104.36.82.12:4179): protocol: rpc: dial tcp 104.36.82.12:4179: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703825932 (27h15m59.484759308s) [noise=true , address=104.36.82.41:4786 , piece_id=F5CD6XOUJSE4TMCKTAAV2QWL5G4CRMQTPDPOC7MIAATBOUKARR3A, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q, address:104.36.82.41:4786): protocol: rpc: dial tcp 104.36.82.41:4786: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703895939 (27h15m59.484829315s) [noise=true , address=104.36.82.47:4533 , piece_id=V2MWRXPPEF4E4A4NYUXPMCMQ4W4YFMIXF5JWDY4BCGHC2IMFGMXQ, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: rpc: dial tcp 104.36.82.47:4533: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703928400 (27h15m59.484861775s) [noise=true , address=104.36.82.55:4196 , piece_id=MLT4HSX753OPJAJN23UVXJHAVRE7SN5ARTFAEPACKCVBZ5RLKQWQ, seg_pos={0 0}, upload=25999]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: write tcp 159.223.143.39:45994->104.36.82.55:4196: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:43.703955921 (27h15m59.484889289s) [seg_pos={0 0}, upload=25999]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 01:07:43.740523651 (27h15m59.52145702s) [upload=25999]: Done uploading segments... Jul 28 01:07:43.740550130 (27h15m59.521483498s) [upload=25999]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.040167208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........XX...........X...X...X.............................] duration: 412.512587ms FAIL (2025-07-28 01:07:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 412.547418ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X...........X...................X....XX........X..X.......] duration: 404.363436ms FAIL (2025-07-28 01:07:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 404.394576ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 505.319593ms FAIL (2025-07-28 01:07:24 UTC) orbiter warning: uplink: too many requests Duration: 505.354206ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X...........X...................X....XX........X..X......] duration: 440.021981ms FAIL (2025-07-28 01:06:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 440.054027ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 240.36787ms FAIL (2025-07-28 01:06:46 UTC) orbiter warning: uplink: too many requests Duration: 240.396113ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027296853s FAIL (2025-07-28 01:06:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 3m59.55730432s) [noise=true , address=104.36.82.11:4118 , piece_id=GZIMVQZV66TXCY5XYNWWT2IK7T4BVVTRMSQVXAYFHI4E7YLIMWVQ, seg_pos={0 0}, upload=25981]: Uploading piece... Jul 28 01:05:43.776690877 (27h13m59.557624261s) [noise=true , address=104.36.82.42:4824 , piece_id=6CZJ2BRFGBA43RRLQYDW5BE54MKOEKQIDVYIDSV5T2ROX4SHF5RQ, seg_pos={0 0}, upload=25981]: Uploading piece... Jul 28 01:05:43.777029776 (27h13m59.557963161s) [noise=true , address=104.36.82.40:4963 , piece_id=YVKOEQJFQOGJ3YOA27OZEQ332DLDGZXKKMTBIOAQKVEGOC36XBBQ, seg_pos={0 0}, upload=25981]: Uploading piece... Jul 28 01:05:43.777403794 (27h13m59.558337173s) [noise=true , address=104.36.82.56:4248 , piece_id=HECOWX4AKIJBWGXBHF2VBKRQ4ZZBVZV2LCFAHJZANRWZRWGESCEA, seg_pos={0 0}, upload=25981]: Uploading piece... Jul 28 01:05:43.777682261 (27h13m59.55861564s) [noise=true , address=104.36.82.53:4133 , piece_id=Y66RGJG4AJDGCE34MF6C7F5HYI2GZ4SKWMNVIR46FDST7L3LW6ZQ, seg_pos={0 0}, upload=25981]: Uploading piece... Jul 28 01:05:43.777964064 (27h13m59.558897442s) [noise=true , address=104.36.82.55:4224 , piece_id=WWMTX5FJMSYRFNGZQJ3IKFQPNLYCFBULTDEHM5ANK4J3GPNMMP5A, seg_pos={0 0}, upload=25981]: Uploading piece... Jul 28 01:05:43.778250912 (27h13m59.559184291s) [noise=true , address=104.36.82.21:4489 , piece_id=Q5ISORDO5AAQNII7WDSFBPU35XVWPKFFRWNXKJHH4GGH42TCXSSQ, seg_pos={0 0}, upload=25981]: Uploading piece... Jul 28 01:05:43.778640078 (27h13m59.559573458s) [noise=true , address=104.36.82.2:4744 , piece_id=AX56HPICUR42NMEMV3MS5AXQWTIEU2FLK7BWKFC7LTXRZ3YYYLMA, seg_pos={0 0}, upload=25981]: Uploading piece... Jul 28 01:05:44.004656624 (27h13m59.785590022s) [upload=25981]: Next returned nil segment Jul 28 01:05:44.004680366 (27h13m59.785613755s) [upload=25981]: Waiting for error group managing segments... Jul 28 01:05:48.183757927 (27h14m3.964691295s) [noise=true , address=79.127.201.215:4358 , piece_id=6DJXH2IFBEOBIBB3T4WBUAAMPONQHUHOPZ3FR3AZ3RCGXV6L3T7Q, seg_pos={0 0}, upload=25981]: Done uploading piece. err: <nil> Jul 28 01:06:43.619895475 (27h14m59.400828857s) [noise=true , address=104.36.82.7:4919 , piece_id=ZKVY2D6RGXMI45RVYUTFZHECSO5C6FVFACWLTOETSOCC4VP2JW2A, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:182cMUjmC3Ku6z4KucQpqTS8NcKz7VJKeXt9tFYPnUtDUj6A2w, address:104.36.82.7:4919): protocol: rpc: dial tcp 104.36.82.7:4919: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.619983575 (27h14m59.400916951s) [noise=true , address=104.36.82.4:4827 , piece_id=FNQLNS2DGZRKA4IVTJRLAPPMY4OST7XS4IPOQN4WDGQJ4CSPWABQ, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:1VXFTn1urnJ2qkUcDc5785hJ8L6bNVQTtgcgNeWLaZ9wEkKYVx, address:104.36.82.4:4827): protocol: rpc: dial tcp 104.36.82.4:4827: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620048238 (27h14m59.400981614s) [noise=true , address=104.36.82.54:4168 , piece_id=CFUL6ORAPBEB5L4K3ETG4VPWQDXVUIFJ6R44DLUMVAGJ4DOLA5BA, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: rpc: dial tcp 104.36.82.54:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620118748 (27h14m59.401052128s) [noise=true , address=104.36.82.5:4851 , piece_id=W6KPSWXBURE3XQ4J2Q6RRRSGY3MWFDPS5KRFL43JNZ2S6C6MNJXA, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:104.36.82.5:4851): protocol: rpc: dial tcp 104.36.82.5:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620266121 (27h14m59.401199502s) [noise=true , address=104.36.82.56:4248 , piece_id=HECOWX4AKIJBWGXBHF2VBKRQ4ZZBVZV2LCFAHJZANRWZRWGESCEA, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:1QpYBQ2PfiAwvt5EmFvjBnKgBhwnuyF6b77qJocpNe1YqSzP77, address:104.36.82.56:4248): protocol: rpc: dial tcp 104.36.82.56:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620337541 (27h14m59.401270916s) [noise=true , address=104.36.82.53:4133 , piece_id=Y66RGJG4AJDGCE34MF6C7F5HYI2GZ4SKWMNVIR46FDST7L3LW6ZQ, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: rpc: dial tcp 104.36.82.53:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620376591 (27h14m59.401309976s) [noise=true , address=104.36.82.11:4118 , piece_id=GZIMVQZV66TXCY5XYNWWT2IK7T4BVVTRMSQVXAYFHI4E7YLIMWVQ, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE, address:104.36.82.11:4118): protocol: rpc: dial tcp 104.36.82.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620392139 (27h14m59.401325515s) [noise=true , address=104.36.82.42:4824 , piece_id=6CZJ2BRFGBA43RRLQYDW5BE54MKOEKQIDVYIDSV5T2ROX4SHF5RQ, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:12QTjnomQYFETHbaJBWxWEPo5yweArk68bKPeooZXpRZrpw1ACq, address:104.36.82.42:4824): protocol: rpc: dial tcp 104.36.82.42:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620438057 (27h14m59.401371432s) [noise=true , address=104.36.82.55:4224 , piece_id=WWMTX5FJMSYRFNGZQJ3IKFQPNLYCFBULTDEHM5ANK4J3GPNMMP5A, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: rpc: dial tcp 104.36.82.55:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620443457 (27h14m59.401376833s) [noise=true , address=104.36.82.21:4489 , piece_id=Q5ISORDO5AAQNII7WDSFBPU35XVWPKFFRWNXKJHH4GGH42TCXSSQ, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:12fao48Sd4XTApMqhudk7rT4pk6FssZRoqNdoUnrkL2b8cyExc5, address:104.36.82.21:4489): protocol: rpc: dial tcp 104.36.82.21:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620478296 (27h14m59.401411671s) [noise=true , address=104.36.82.12:4179 , piece_id=YVHNADSNOHSIJJ2IXED3UMCAEUK4XETKEXFUQ423YCCHKHJP3XEQ, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:129pP9f3rFfTvP6MhG894N7WHuJRZP8aWhzJf1zBwZ76wbzVbZa, address:104.36.82.12:4179): protocol: rpc: dial tcp 104.36.82.12:4179: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620516889 (27h14m59.401450268s) [noise=true , address=104.36.82.46:4502 , piece_id=53GUHKXC7KWU7ATVHSTGWJRXGSUS3IHPDMV2XBU5CBX3SQJ2QXHA, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620551199 (27h14m59.401484576s) [noise=true , address=104.36.82.40:4963 , piece_id=YVKOEQJFQOGJ3YOA27OZEQ332DLDGZXKKMTBIOAQKVEGOC36XBBQ, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm, address:104.36.82.40:4963): protocol: write tcp 159.223.143.39:35404->104.36.82.40:4963: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620558362 (27h14m59.401491724s) [noise=true , address=104.36.82.2:4744 , piece_id=AX56HPICUR42NMEMV3MS5AXQWTIEU2FLK7BWKFC7LTXRZ3YYYLMA, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH, address:104.36.82.2:4744): protocol: write tcp 159.223.143.39:57418->104.36.82.2:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620589045 (27h14m59.401522422s) [noise=true , address=104.36.82.10:4095 , piece_id=S4NLRJUR3XCHARIXVEQSPYIQXRU45TIPYLNQJGFRMS325YNVYHAA, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:12gs9LRE3oUp1jupCw5uDvRtkNcahbq2d4VTYQsdKr6atk6vGJx, address:104.36.82.10:4095): protocol: rpc: dial tcp 104.36.82.10:4095: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620607321 (27h14m59.401540683s) [noise=true , address=104.36.82.15:4283 , piece_id=HZFIUJA6C7P2DK6AK3NETLAWGMRNCHSIO2JGV6YZ2WWSXWQSGHWA, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU, address:104.36.82.15:4283): protocol: rpc: dial tcp 104.36.82.15:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620615270 (27h14m59.401548631s) [noise=true , address=104.36.82.11:4128 , piece_id=IPSCUAAY5PUYGHG2A6LY4WUC5OZDIMGNZ57TDYOD4GKEOL5APJZQ, seg_pos={0 0}, upload=25981]: Done uploading piece. err: ecclient: upload failed (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS, address:104.36.82.11:4128): protocol: rpc: dial tcp 104.36.82.11:4128: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:43.620630508 (27h14m59.401563883s) [seg_pos={0 0}, upload=25981]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:06:43.645360423 (27h14m59.426293803s) [upload=25981]: Done uploading segments... Jul 28 01:06:43.645388922 (27h14m59.426322308s) [upload=25981]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.027316284s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........XX...........X...X...X............................] duration: 554.014499ms FAIL (2025-07-28 01:06:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 554.043119ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 319.005079ms FAIL (2025-07-28 01:06:31 UTC) orbiter warning: uplink: too many requests Duration: 319.04103ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X...........X...................X....XX........X..X.....] duration: 457.788819ms FAIL (2025-07-28 01:05:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 457.833687ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.028827061s FAIL (2025-07-28 01:05:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 291 (27h12m59.466592667s) [noise=true , address=104.36.82.11:4131 , piece_id=GGYJ4W6EQXMZRR37GZZXK3ALCQWJ6CZ4KAKECRCX5UHZXYAFSLBQ, seg_pos={0 0}, upload=25967]: Uploading piece... Jul 28 01:04:43.685950657 (27h12m59.466884035s) [noise=true , address=104.36.82.9:4022 , piece_id=AQJ4NH4VUTMEUVRFAQ6HVNA4VHDT35MLX3EVLHNYLIMYAY7F52FQ, seg_pos={0 0}, upload=25967]: Uploading piece... Jul 28 01:04:43.686262395 (27h12m59.467195774s) [noise=true , address=104.36.82.5:4849 , piece_id=TUL5IGL7DXZJOCN43RGXXVERSAVVFS2HJPDZME766IOV6PX65MTQ, seg_pos={0 0}, upload=25967]: Uploading piece... Jul 28 01:04:43.686534914 (27h12m59.46746829s) [noise=true , address=104.36.82.47:4535 , piece_id=GQZ723TBTDD5FW7WOWDBITOMJUL7REZJ4WF6S3O4PRQZBLNXKLRA, seg_pos={0 0}, upload=25967]: Uploading piece... Jul 28 01:04:43.686805520 (27h12m59.467738895s) [noise=true , address=104.36.82.16:4322 , piece_id=LDLQHSRWHCANCTF5DTRBMVCA5VW5VCCLKPHVICLYY7WB5HXSZAYA, seg_pos={0 0}, upload=25967]: Uploading piece... Jul 28 01:04:43.689481282 (27h12m59.470414668s) [noise=true , address=104.36.82.52:4114 , piece_id=WB2CVAX6LYICQT43VMLANVX44KXJKPZLJO7CKB5NTVUN5EZBYJHA, seg_pos={0 0}, upload=25967]: Uploading piece... Jul 28 01:04:43.689947939 (27h12m59.470881324s) [noise=true , address=104.36.82.14:4240 , piece_id=VDHV6XOASQ2MER65K7I7Z7PT7KBE3YCKCZBQVWOZT3HLNBMTLLKQ, seg_pos={0 0}, upload=25967]: Uploading piece... Jul 28 01:04:43.772221800 (27h12m59.553155184s) [upload=25967]: Next returned nil segment Jul 28 01:04:43.772246555 (27h12m59.553179943s) [upload=25967]: Waiting for error group managing segments... Jul 28 01:04:43.831212711 (27h12m59.612146093s) [noise=true , address=79.127.219.38:4018 , piece_id=FM22V5ZRYHSOFLDF7WNILCBADNRFSH45654N7YC6XCTTVXIXUPZA, seg_pos={0 0}, upload=25967]: Done uploading piece. err: <nil> Jul 28 01:05:43.520683112 (27h13m59.301616501s) [noise=true , address=104.36.82.5:4849 , piece_id=TUL5IGL7DXZJOCN43RGXXVERSAVVFS2HJPDZME766IOV6PX65MTQ, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1, address:104.36.82.5:4849): protocol: write tcp 159.223.143.39:40282->104.36.82.5:4849: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.520770066 (27h13m59.301703453s) [noise=true , address=104.36.82.52:4114 , piece_id=WB2CVAX6LYICQT43VMLANVX44KXJKPZLJO7CKB5NTVUN5EZBYJHA, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:12PXReFMGqAhCAp5BSE3vsJdfRAUathsJLdB8Guz7DzW6UHmATM, address:104.36.82.52:4114): protocol: rpc: dial tcp 104.36.82.52:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.520788794 (27h13m59.301722158s) [noise=true , address=104.36.82.14:4240 , piece_id=VDHV6XOASQ2MER65K7I7Z7PT7KBE3YCKCZBQVWOZT3HLNBMTLLKQ, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:13SaC5StbguPT3abAT9oTX3mZAyGhJcpAA6b7ueyiDFHa8CCEo, address:104.36.82.14:4240): protocol: write tcp 159.223.143.39:49540->104.36.82.14:4240: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.520861796 (27h13m59.301795158s) [noise=true , address=104.36.82.42:4803 , piece_id=DXSNRXIMDXGDQZ3S3T3L7HGQ4QO3SXGZAWHDZ372RXHWXGUH23HQ, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:12GFk3GkhB88UKQdZM8vZ9VLNuuYDrS5ThZjVAdbL2CtTQVMDEB, address:104.36.82.42:4803): protocol: write tcp 159.223.143.39:36172->104.36.82.42:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.520872278 (27h13m59.301805665s) [noise=true , address=104.36.82.12:4163 , piece_id=6FQY7TNMCJVOZANV42GI6N2T7SLXZLMGKPHMK2ISKADAPPGOHTIA, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC, address:104.36.82.12:4163): protocol: rpc: dial tcp 104.36.82.12:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.520935875 (27h13m59.301869253s) [noise=true , address=104.36.82.55:4200 , piece_id=IG2MYAD5FQHWP226GJ2JHJZDGIUE6KHSUNPBLT7IMIN2K3F3POFA, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: rpc: dial tcp 104.36.82.55:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.520982913 (27h13m59.301916295s) [noise=true , address=104.36.82.15:4265 , piece_id=75J5W4NGGGTRWVSGAFEZD3UGHEOFQAXOV5UFN2OUEKJMQVXG3KPQ, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:1NscbwYiKarBdony9TwdccuEKg7f4VXEXerFfWmCiat4RKdXFi, address:104.36.82.15:4265): protocol: rpc: dial tcp 104.36.82.15:4265: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.521000854 (27h13m59.301934216s) [noise=true , address=104.36.82.47:4535 , piece_id=GQZ723TBTDD5FW7WOWDBITOMJUL7REZJ4WF6S3O4PRQZBLNXKLRA, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: write tcp 159.223.143.39:50486->104.36.82.47:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.521069763 (27h13m59.302003131s) [noise=true , address=104.36.82.6:4881 , piece_id=ZIJZVZ6KC2OZAK5HVT6YO4XDPMBLRXCW27SQZGSY6E6MM6OEB4VA, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:124bCrEcbY1h4ZKxPeiAxX9v1LKhRJrMuYbKmw6nuYEwDZngm2r, address:104.36.82.6:4881): protocol: rpc: dial tcp 104.36.82.6:4881: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.521076814 (27h13m59.302010176s) [noise=true , address=104.36.82.7:4915 , piece_id=W4JPSDWDB6MAGTQVLPNBOXUTSGX54CORMENUWSD2D46L2BFTOXZQ, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:104.36.82.7:4915): protocol: rpc: dial tcp 104.36.82.7:4915: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.521133451 (27h13m59.302066829s) [noise=true , address=104.36.82.56:4263 , piece_id=L444LYAYKC552IJSWGU4GGPHPPALPE6H4IIAV6CUR2UWTUWRRKGQ, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:12tLW4r8AQKLCbSWZoNUVi5v2YtF8NthUmX81TLQn8G8TotuByG, address:104.36.82.56:4263): protocol: rpc: dial tcp 104.36.82.56:4263: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.521169797 (27h13m59.302103182s) [noise=true , address=104.36.82.54:4176 , piece_id=I4UMNKI7VADPCDVV3SCJ2EXQBJL7LINKPDEG7EMDOOIPNRFK5E7Q, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: write tcp 159.223.143.39:37236->104.36.82.54:4176: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.521230177 (27h13m59.302163543s) [noise=true , address=104.36.82.53:4136 , piece_id=RRJVAT7I7UKD6ECH2SG3E35XOGXLVZBET2JU3DMQLWPPYJ2KOG7Q, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:104.36.82.53:4136): protocol: write tcp 159.223.143.39:58908->104.36.82.53:4136: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.521248348 (27h13m59.30218171s) [noise=true , address=104.36.82.11:4131 , piece_id=GGYJ4W6EQXMZRR37GZZXK3ALCQWJ6CZ4KAKECRCX5UHZXYAFSLBQ, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 159.223.143.39:44760->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.521293704 (27h13m59.302227084s) [noise=true , address=104.36.82.16:4322 , piece_id=LDLQHSRWHCANCTF5DTRBMVCA5VW5VCCLKPHVICLYY7WB5HXSZAYA, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:104.36.82.16:4322): protocol: rpc: dial tcp 104.36.82.16:4322: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.521312050 (27h13m59.302245432s) [noise=true , address=104.36.82.52:4094 , piece_id=7RZ4BPIRLV27VZGP2XE2KU7GOSBBIGEDNXLYFM4TY62JM3HFV3EQ, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: rpc: dial tcp 104.36.82.52:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.521349870 (27h13m59.302283248s) [noise=true , address=104.36.82.9:4022 , piece_id=AQJ4NH4VUTMEUVRFAQ6HVNA4VHDT35MLX3EVLHNYLIMYAY7F52FQ, seg_pos={0 0}, upload=25967]: Done uploading piece. err: ecclient: upload failed (node:1XGBWwhqJ324UTpv3b1bNsGVUjJSLptvqV6dHrYJsto8KbDvYU, address:104.36.82.9:4022): protocol: rpc: dial tcp 104.36.82.9:4022: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:43.521389805 (27h13m59.302323189s) [seg_pos={0 0}, upload=25967]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:05:43.546006762 (27h13m59.326940144s) [upload=25967]: Done uploading segments... Jul 28 01:05:43.546038225 (27h13m59.326971611s) [upload=25967]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.028845684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 49.377134ms FAIL (2025-07-28 01:05:41 UTC) orbiter warning: uplink: too many requests Duration: 49.403991ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........XX...........X...X...X...........................] duration: 444.873109ms FAIL (2025-07-28 01:05:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.905223ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X...........X...................X....XX........X..X....] duration: 593.422971ms FAIL (2025-07-28 01:04:51 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 593.50437ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [.XXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.040421777s FAIL (2025-07-28 01:04:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== ue , address=79.127.201.215:4358 , piece_id=AM5EIQPZE6PVR5MPFVF37FIEVZ5EFTBRGL5EACGHIVMUMKSADJLA, seg_pos={0 0}, upload=25953]: Uploading piece... Jul 28 01:03:43.510846996 (27h11m59.291780412s) [noise=true , address=104.36.82.15:4291 , piece_id=3IJZW2VUSQLRU3MMRUJ2H6URDXHUC57PUCG6X2O5G33TBX3R76OQ, seg_pos={0 0}, upload=25953]: Uploading piece... Jul 28 01:03:43.511671301 (27h11m59.292604691s) [noise=true , address=104.36.82.13:4211 , piece_id=5N6SM64SL4COHOII3WOXCH7MBX6W4S5ABCKBWYCSCMFKALMRNWQA, seg_pos={0 0}, upload=25953]: Uploading piece... Jul 28 01:03:43.512850781 (27h11m59.293784164s) [noise=true , address=104.36.82.49:4981 , piece_id=BXGI52UV77OWKXI4JF6ARMJKWTEGYJRSIL4JVV5ITEPUTZ7UCZ3A, seg_pos={0 0}, upload=25953]: Uploading piece... Jul 28 01:03:43.513827562 (27h11m59.294760947s) [noise=true , address=104.36.82.55:4216 , piece_id=H6KZJTWJOII5MQVHVXNLOC5VI6PSYFDMEGEO4I5ENZXNIMBMSGGQ, seg_pos={0 0}, upload=25953]: Uploading piece... Jul 28 01:03:43.514786419 (27h11m59.295719801s) [noise=true , address=104.36.82.12:4166 , piece_id=HBSGHQYAGO6G3B7T3K7T2FRQSRXUDGQOMCJUYKSCDLQZGHYLMXJQ, seg_pos={0 0}, upload=25953]: Uploading piece... Jul 28 01:03:43.714971685 (27h11m59.495905063s) [upload=25953]: Next returned nil segment Jul 28 01:03:43.714993386 (27h11m59.495926757s) [upload=25953]: Waiting for error group managing segments... Jul 28 01:03:45.481375537 (27h12m1.262308911s) [noise=true , address=79.127.201.215:4358 , piece_id=AM5EIQPZE6PVR5MPFVF37FIEVZ5EFTBRGL5EACGHIVMUMKSADJLA, seg_pos={0 0}, upload=25953]: Done uploading piece. err: <nil> Jul 28 01:04:43.410821646 (27h12m59.191755021s) [noise=true , address=104.36.82.16:4301 , piece_id=27DERMWGUCNNYROGABOOTVQKFNOIZCT3JAOYRYIAKM66KU5B45PQ, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG, address:104.36.82.16:4301): protocol: rpc: dial tcp 104.36.82.16:4301: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.410931958 (27h12m59.191865333s) [noise=true , address=104.36.82.49:4981 , piece_id=BXGI52UV77OWKXI4JF6ARMJKWTEGYJRSIL4JVV5ITEPUTZ7UCZ3A, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:1u49RC6mHSy43oo9REosyNVdFB3jKk98A6hH4j9qgCQaF5qPSv, address:104.36.82.49:4981): protocol: rpc: dial tcp 104.36.82.49:4981: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411003698 (27h12m59.191937073s) [noise=true , address=104.36.82.55:4216 , piece_id=H6KZJTWJOII5MQVHVXNLOC5VI6PSYFDMEGEO4I5ENZXNIMBMSGGQ, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411113785 (27h12m59.19204716s) [noise=true , address=104.36.82.12:4166 , piece_id=HBSGHQYAGO6G3B7T3K7T2FRQSRXUDGQOMCJUYKSCDLQZGHYLMXJQ, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411294594 (27h12m59.192227956s) [noise=true , address=104.36.82.53:4156 , piece_id=M27UI5QDAAVFUEWYADWX2XQULPSISVQPA24IOJYHTLJZ5TYTA6EA, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 159.223.143.39:51984->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411378881 (27h12m59.192312246s) [noise=true , address=104.36.82.54:4159 , piece_id=3SKXWKUQGI467XCKDNXAY3BRZMTSYOWOZZUR7XZ6PDDYBN3KCXQQ, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:104.36.82.54:4159): protocol: write tcp 159.223.143.39:53556->104.36.82.54:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411441109 (27h12m59.192374486s) [noise=true , address=104.36.82.44:4901 , piece_id=2UYO2E5NM3LKUHGHVICFGPXIYXKGOCRR6XPQT56CIA7YCOTQGSFQ, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: write tcp 159.223.143.39:48434->104.36.82.44:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411508305 (27h12m59.192441681s) [noise=true , address=104.36.82.6:4887 , piece_id=QK3Q23V4LEDL4FGZELQWFBVMF2NXYTR3GDNQZYHXLSDTTJ773PDA, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7, address:104.36.82.6:4887): protocol: write tcp 159.223.143.39:53290->104.36.82.6:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411594100 (27h12m59.192527461s) [noise=true , address=104.36.82.10:4083 , piece_id=KMYYNTMG2ZQED47YCNQLS23DJX7ULYDPIH6PAAAVNPR4WEKFEJ6Q, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh, address:104.36.82.10:4083): protocol: write tcp 159.223.143.39:42702->104.36.82.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411669575 (27h12m59.192602938s) [noise=true , address=104.36.82.46:4501 , piece_id=A7B2MA6ALLHLMWJUSEIYF5OVITEBF3E2QD2BVP67XJMFIQRGEIWQ, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: write tcp 159.223.143.39:57892->104.36.82.46:4501: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411679695 (27h12m59.192613058s) [noise=true , address=104.36.82.56:4253 , piece_id=4OSFPEHMTYW6V5OJA7HB2JGTNWYKRQN36NHLWJNN6JX73IMDYFPQ, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: write tcp 159.223.143.39:38828->104.36.82.56:4253: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411731222 (27h12m59.192664601s) [noise=true , address=104.36.82.16:4322 , piece_id=I3R3LGYBD7I3M24YCQ2CEW4PFUVNUTQOQ4QFNACJM5M2RN4G4VUA, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3, address:104.36.82.16:4322): protocol: write tcp 159.223.143.39:43798->104.36.82.16:4322: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411787793 (27h12m59.192721175s) [noise=true , address=104.36.82.13:4211 , piece_id=5N6SM64SL4COHOII3WOXCH7MBX6W4S5ABCKBWYCSCMFKALMRNWQA, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:12dANoYTygpcpWfCcAJtsnVBFWSekLhDjheAXnsaYMChRYAm2qc, address:104.36.82.13:4211): protocol: write tcp 159.223.143.39:41290->104.36.82.13:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411813294 (27h12m59.192746668s) [noise=true , address=104.36.82.14:4226 , piece_id=XN4RSIGMY34GTDZQJQ272HP4RIQZ4GOA5VBFHUKEY2TWJLXYJKLA, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt, address:104.36.82.14:4226): protocol: write tcp 159.223.143.39:33784->104.36.82.14:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411888160 (27h12m59.192821535s) [noise=true , address=104.36.82.15:4291 , piece_id=3IJZW2VUSQLRU3MMRUJ2H6URDXHUC57PUCG6X2O5G33TBX3R76OQ, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:4291): protocol: write tcp 159.223.143.39:40296->104.36.82.15:4291: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411895475 (27h12m59.192828841s) [noise=true , address=104.36.82.41:4778 , piece_id=3XBZSDMDWNXDS5W3UAUQDBAAMF6UAZTZREVJDRSBLS2EFUVALKOA, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV, address:104.36.82.41:4778): protocol: write tcp 159.223.143.39:60598->104.36.82.41:4778: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.411955115 (27h12m59.192888492s) [noise=true , address=104.36.82.21:4475 , piece_id=CRQHIVRWVAMKQPODPS5FXIZJH5ADQFWWGKVPEFYOVUBDDFUDTYNA, seg_pos={0 0}, upload=25953]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:104.36.82.21:4475): protocol: write tcp 159.223.143.39:51578->104.36.82.21:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:04:43.412012875 (27h12m59.192946259s) [seg_pos={0 0}, upload=25953]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:04:43.448868406 (27h12m59.229801772s) [upload=25953]: Done uploading segments... Jul 28 01:04:43.448894203 (27h12m59.229827578s) [upload=25953]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.040440441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............XX...........X...X...X..........................] duration: 536.435671ms FAIL (2025-07-28 01:04:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 536.632767ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [..XXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033461077s FAIL (2025-07-28 01:03:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ZWPKITA, seg_pos={0 0}, upload=25935]: Uploading piece... Jul 28 01:02:43.396037734 (27h10m59.176971123s) [noise=true , address=104.36.82.54:4175 , piece_id=S6IIWWGWSELS2BLKGXLPQFI4BD2EWGGOWSPTSGM3EFTT6FMGLVEQ, seg_pos={0 0}, upload=25935]: Uploading piece... Jul 28 01:02:43.396547161 (27h10m59.177480554s) [noise=true , address=104.36.82.20:4451 , piece_id=BCSXMCULVIVLFSKL5OSY4EDHKXBTCT4EKAZQDG3CLBDGADF7RH2Q, seg_pos={0 0}, upload=25935]: Uploading piece... Jul 28 01:02:43.398677307 (27h10m59.179610695s) [noise=true , address=104.36.82.53:4126 , piece_id=SMSPVFPDQBPFRORXGE3BEFBPH4B54IMXXO5CBABXG4FOHAUJVEZQ, seg_pos={0 0}, upload=25935]: Uploading piece... Jul 28 01:02:43.403437510 (27h10m59.184370898s) [noise=true , address=104.36.82.7:4937 , piece_id=DQD57ERPE7UHTNUGL4X3MZGVVJG3WSPK44QATGFWC7PPSZ263XGA, seg_pos={0 0}, upload=25935]: Uploading piece... Jul 28 01:02:43.416001519 (27h10m59.196934905s) [noise=true , address=104.36.82.50:4035 , piece_id=D2KJESLFD6PNZV2JLT5G7Z73DMRGTTWZZN647E53DCNUYM57WDSA, seg_pos={0 0}, upload=25935]: Uploading piece... Jul 28 01:02:43.418281567 (27h10m59.199214947s) [noise=true , address=104.36.82.55:4218 , piece_id=TEBEKNOVN27OLYK6X57PRHK6VVERXDLPDMU22T7OFRNIO4GBJI2A, seg_pos={0 0}, upload=25935]: Uploading piece... Jul 28 01:02:43.421240227 (27h10m59.202173646s) [noise=true , address=104.36.82.9:4010 , piece_id=RILTJTWCWG67MSYNELJA7VJZCS2KDZFLHRNPAZIR4QNYQU3PWCTA, seg_pos={0 0}, upload=25935]: Uploading piece... Jul 28 01:02:43.933736900 (27h10m59.714670268s) [upload=25935]: Next returned nil segment Jul 28 01:02:43.933757188 (27h10m59.71469055s) [upload=25935]: Waiting for error group managing segments... Jul 28 01:02:44.096971993 (27h10m59.877905374s) [noise=true , address=64.49.13.138:4482 , piece_id=SJ7I4FJRRJUINIT2PF6B4OI6SZNPFBLEBFPPQITIDK625XZ7JL2A, seg_pos={0 0}, upload=25935]: Done uploading piece. err: <nil> Jul 28 01:02:44.430734497 (27h11m0.211667877s) [noise=true , address=64.49.13.138:4483 , piece_id=2SAECQS3O7SSDXJ7GUIU5TYLBWUYZIGBY5EU7ULZVYCBGXYRTG7A, seg_pos={0 0}, upload=25935]: Done uploading piece. err: <nil> Jul 28 01:03:43.313557816 (27h11m59.094491193s) [noise=true , address=104.36.82.56:4239 , piece_id=UGZNKV2JR5LHYOHJHWOEKRV6W5EMXSXG5U6KG6CADO6V77JGTHLQ, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: write tcp 159.223.143.39:53136->104.36.82.56:4239: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.313623387 (27h11m59.094556753s) [noise=true , address=104.36.82.21:4493 , piece_id=BVJ5WICEGW3E7XPFUAKE24PGX2J4HQBQ3Y3V6IGYOG27H67YV3KA, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:12PUjzJu7JJ4q77xnkBEZtPV2bwg5UJtMBfkgnjuwdAHc7Ff5UD, address:104.36.82.21:4493): protocol: write tcp 159.223.143.39:39260->104.36.82.21:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.313678062 (27h11m59.094611439s) [noise=true , address=104.36.82.13:4200 , piece_id=XIYMIYTA46FXVOOEPNOXJE5O5BE6K3B2AK7TF5B7IEJ6XWLHYY5Q, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:12dfLumooVsDsWXqi5FQMDutkcexM9A5Kmu4Vo9s1TqfWxv5ozH, address:104.36.82.13:4200): protocol: write tcp 159.223.143.39:36894->104.36.82.13:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.313735340 (27h11m59.094668718s) [noise=true , address=104.36.82.42:4828 , piece_id=676ZHRIEOETUYKWFEGH2BHUTBX5RXZGBQ4RMA4DG75PNQDF4WDAA, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:1cX7FtKC8KMopnX9zvkT6fYdnutK84cFjLW6xMwpXuGsqpiN8k, address:104.36.82.42:4828): protocol: write tcp 159.223.143.39:33930->104.36.82.42:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.313784205 (27h11m59.094717581s) [noise=true , address=104.36.82.15:4268 , piece_id=AECDPKTCY5ZNEILUXWFXSDVLREQ7HX5GZBINJ2MXZEGS5G2QOFCA, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:112ocg8g8hf2huUskC1Kty3z51wXc8VuNUgARTFjKZxsF6RFFy, address:104.36.82.15:4268): protocol: rpc: dial tcp 104.36.82.15:4268: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.314029982 (27h11m59.094963383s) [noise=true , address=104.36.82.55:4218 , piece_id=TEBEKNOVN27OLYK6X57PRHK6VVERXDLPDMU22T7OFRNIO4GBJI2A, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 159.223.143.39:40240->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.314139104 (27h11m59.09507247s) [noise=true , address=104.36.82.7:4937 , piece_id=DQD57ERPE7UHTNUGL4X3MZGVVJG3WSPK44QATGFWC7PPSZ263XGA, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:104.36.82.7:4937): protocol: write tcp 159.223.143.39:47334->104.36.82.7:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.314193562 (27h11m59.095126941s) [noise=true , address=104.36.82.14:4253 , piece_id=4IRKTKUQUQFNP422TSQQAFGZ3CP2ME672D6EIPUFJDWUVV65RL5A, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14, address:104.36.82.14:4253): protocol: write tcp 159.223.143.39:54748->104.36.82.14:4253: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.314269475 (27h11m59.095202853s) [noise=true , address=104.36.82.22:4514 , piece_id=A4WDU4SH3XVERH6PAWKFW667KYTEIG5RZB2IR7VHC3XEKZWPKITA, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: write tcp 159.223.143.39:45644->104.36.82.22:4514: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.314336761 (27h11m59.095270127s) [noise=true , address=104.36.82.53:4126 , piece_id=SMSPVFPDQBPFRORXGE3BEFBPH4B54IMXXO5CBABXG4FOHAUJVEZQ, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: rpc: dial tcp 104.36.82.53:4126: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.314450022 (27h11m59.095383396s) [noise=true , address=104.36.82.49:4005 , piece_id=XEXH5FSR6A5LTQZNFLYYXLQIVED4NJE5X3NFFUA3RJCA3275GA5Q, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.314528975 (27h11m59.095462362s) [noise=true , address=104.36.82.47:4542 , piece_id=PVJYODLUYR4CJHJCO3345RSMSUQ3HWGCH62DAIEZVVCNRRYUB4DQ, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: write tcp 159.223.143.39:42756->104.36.82.47:4542: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.314602076 (27h11m59.095535454s) [noise=true , address=104.36.82.50:4035 , piece_id=D2KJESLFD6PNZV2JLT5G7Z73DMRGTTWZZN647E53DCNUYM57WDSA, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: write tcp 159.223.143.39:43314->104.36.82.50:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.314680105 (27h11m59.095613486s) [noise=true , address=104.36.82.20:4451 , piece_id=BCSXMCULVIVLFSKL5OSY4EDHKXBTCT4EKAZQDG3CLBDGADF7RH2Q, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:12fWecKRc63WWDEqfHomHdVKVaH3naVgZx6nhLZDJ4JnL3CfPQ, address:104.36.82.20:4451): protocol: write tcp 159.223.143.39:34778->104.36.82.20:4451: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.314780971 (27h11m59.095714353s) [noise=true , address=104.36.82.9:4010 , piece_id=RILTJTWCWG67MSYNELJA7VJZCS2KDZFLHRNPAZIR4QNYQU3PWCTA, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:104.36.82.9:4010): protocol: write tcp 159.223.143.39:54146->104.36.82.9:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:03:43.315668465 (27h11m59.096601845s) [noise=true , address=104.36.82.54:4175 , piece_id=S6IIWWGWSELS2BLKGXLPQFI4BD2EWGGOWSPTSGM3EFTT6FMGLVEQ, seg_pos={0 0}, upload=25935]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:03:43.315699716 (27h11m59.096633101s) [seg_pos={0 0}, upload=25935]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 01:03:43.342237579 (27h11m59.123170958s) [upload=25935]: Done uploading segments... Jul 28 01:03:43.342268586 (27h11m59.123201984s) [upload=25935]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.033494112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............XX...........X...X...X.........................] duration: 438.072513ms FAIL (2025-07-28 01:03:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 438.10668ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X...........X...................X....XX........X..X...] duration: 423.588492ms FAIL (2025-07-28 01:03:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 423.62538ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.............XX...........X...X...X........................] duration: 465.660221ms FAIL (2025-07-28 01:03:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 465.721854ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X...........X...................X....XX........X..X..] duration: 497.322838ms FAIL (2025-07-28 01:03:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 497.354284ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.............XX...........X...X...X.......................] duration: 508.973623ms FAIL (2025-07-28 01:02:49 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 509.005057ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [...XXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030659388s FAIL (2025-07-28 01:02:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 4492850 (27h9m59.065426336s) [noise=true , address=104.36.82.9:4029 , piece_id=ICUW5H5L7LCB4VML7QNYGD7D4N2HPUJKTLNCLUJHRAO2HO6DOI7Q, seg_pos={0 0}, upload=25916]: Uploading piece... Jul 28 01:01:43.284859925 (27h9m59.065793322s) [noise=true , address=104.36.82.15:4260 , piece_id=I3XYJWVXK2F3IJEOLGKGZZBZFSQM5UFYSCPFC35FCGZHWFETFBKQ, seg_pos={0 0}, upload=25916]: Uploading piece... Jul 28 01:01:43.285236162 (27h9m59.066169544s) [noise=true , address=104.36.82.6:4897 , piece_id=2IXCA2BZX2QOE6NBQ33GC7QLWQPNHHODS5N5Q3ULKP5EQPUCSYMQ, seg_pos={0 0}, upload=25916]: Uploading piece... Jul 28 01:01:43.285574943 (27h9m59.066508343s) [noise=true , address=104.36.82.34:4556 , piece_id=C4DJYG3OEDEQ6MT3RSJT2UUMM3NRJ6IMA5KNGPUWFAS5JJMRHCBA, seg_pos={0 0}, upload=25916]: Uploading piece... Jul 28 01:01:43.285924589 (27h9m59.066857975s) [noise=true , address=104.36.82.56:4245 , piece_id=RM5T2KNGJVQLIKOTLU4RBWQ5XNPY6IASMWQ3WUCEWA7ZCVTVW55Q, seg_pos={0 0}, upload=25916]: Uploading piece... Jul 28 01:01:43.286309780 (27h9m59.067243164s) [noise=true , address=104.36.82.51:4054 , piece_id=SF3ZCXF3XAZEEA4XUIF7BCHG6TGI7VIGGWR7VFVD4RMFSSJZM67Q, seg_pos={0 0}, upload=25916]: Uploading piece... Jul 28 01:01:43.286899017 (27h9m59.067832406s) [noise=true , address=104.36.82.21:4473 , piece_id=RMDONFXXTJB4XPNAQKSNEMR7LNIKI5KUPWXTL74FZEPA2CVK7QFQ, seg_pos={0 0}, upload=25916]: Uploading piece... Jul 28 01:01:43.289406060 (27h9m59.070339437s) [noise=true , address=104.36.82.53:4135 , piece_id=MVLNVRE7HDDAEFTFRMIXXVF3LB7JOKSKN4HN2JARQTXPQ6KLUYIA, seg_pos={0 0}, upload=25916]: Uploading piece... Jul 28 01:01:43.501673525 (27h9m59.28260689s) [upload=25916]: Next returned nil segment Jul 28 01:01:43.501699330 (27h9m59.282632718s) [upload=25916]: Waiting for error group managing segments... Jul 28 01:01:44.721920342 (27h10m0.502853706s) [noise=true , address=79.127.201.215:4348 , piece_id=ISYUIFKE7QDPEMBIHQMNNBG6AAAC6MQI2CMLFXGPONET4I2WAPYQ, seg_pos={0 0}, upload=25916]: Done uploading piece. err: <nil> Jul 28 01:02:43.212021461 (27h10m58.992954829s) [noise=true , address=104.36.82.50:4031 , piece_id=I5X7AVXK6HAQG3CETTVSULQHBQGEYQBUQ35NN6PQOBXYJBPTXMMQ, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: rpc: dial tcp 104.36.82.50:4031: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.212093986 (27h10m58.993027352s) [noise=true , address=104.36.82.56:4245 , piece_id=RM5T2KNGJVQLIKOTLU4RBWQ5XNPY6IASMWQ3WUCEWA7ZCVTVW55Q, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:12YnxyH9TZVuVheQpH9NVRfFUeWHJEtV97wyEJafhCzGZSv13Fn, address:104.36.82.56:4245): protocol: rpc: dial tcp 104.36.82.56:4245: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.212163321 (27h10m58.993096688s) [noise=true , address=104.36.82.53:4135 , piece_id=MVLNVRE7HDDAEFTFRMIXXVF3LB7JOKSKN4HN2JARQTXPQ6KLUYIA, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: rpc: dial tcp 104.36.82.53:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.212402141 (27h10m58.993335507s) [noise=true , address=104.36.82.46:4513 , piece_id=COUBEPROKECR5GYVRLLOP3NKDRHBHISPQKMXTW3QXLLFRSM6O5NA, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: rpc: dial tcp 104.36.82.46:4513: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.212493353 (27h10m58.99342672s) [noise=true , address=104.36.82.34:4556 , piece_id=C4DJYG3OEDEQ6MT3RSJT2UUMM3NRJ6IMA5KNGPUWFAS5JJMRHCBA, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: rpc: dial tcp 104.36.82.34:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.212548793 (27h10m58.993482159s) [noise=true , address=104.36.82.21:4473 , piece_id=RMDONFXXTJB4XPNAQKSNEMR7LNIKI5KUPWXTL74FZEPA2CVK7QFQ, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:104.36.82.21:4473): protocol: rpc: dial tcp 104.36.82.21:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.212601717 (27h10m58.993535084s) [noise=true , address=104.36.82.13:4213 , piece_id=XJFROLLAM6IBJNQAJTMORWGYLBQQRGTJFRHBQXM6Q43IT5W2NIQQ, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:12XDcAySWNCL7XoStP5GKzAksN1kTs9L8u4KStgyaiQn3AvLcRM, address:104.36.82.13:4213): protocol: rpc: dial tcp 104.36.82.13:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.212657680 (27h10m58.993591048s) [noise=true , address=104.36.82.51:4054 , piece_id=SF3ZCXF3XAZEEA4XUIF7BCHG6TGI7VIGGWR7VFVD4RMFSSJZM67Q, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v, address:104.36.82.51:4054): protocol: write tcp 159.223.143.39:43236->104.36.82.51:4054: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.212818743 (27h10m58.993752109s) [noise=true , address=104.36.82.2:4739 , piece_id=OE4LZQEJIV2BU64C36V7EG43PKCBPLN4FQJI2NKK5DLZEMINENGA, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:1eEbhCXRGArodr2pffqXQxRR8w6u4c6J6auDbsCwe1NH5Wwycd, address:104.36.82.2:4739): protocol: rpc: dial tcp 104.36.82.2:4739: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.212930205 (27h10m58.99386358s) [noise=true , address=104.36.82.9:4029 , piece_id=ICUW5H5L7LCB4VML7QNYGD7D4N2HPUJKTLNCLUJHRAO2HO6DOI7Q, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:12snsP1y5fbxY8XrDoXihzQrsBxJ5Y38C2jSGhHtV1ukN426eiP, address:104.36.82.9:4029): protocol: rpc: dial tcp 104.36.82.9:4029: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.213018791 (27h10m58.993952174s) [noise=true , address=104.36.82.5:4868 , piece_id=IFCVDNYZLOXWAWT2IJL6PPIRT7S2IDJIZUGOY6YWS4NWPNJTA7PA, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:18Srfics6dFkQeHZ2WF5YkFuLm2719Pw4Xgg9ntk12KnTBcn8v, address:104.36.82.5:4868): protocol: rpc: dial tcp 104.36.82.5:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.213077797 (27h10m58.994011161s) [noise=true , address=104.36.82.15:4260 , piece_id=I3XYJWVXK2F3IJEOLGKGZZBZFSQM5UFYSCPFC35FCGZHWFETFBKQ, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:1aLsgrV3k6vUSavgPkPGA2FF9HR3ryN9bnHutruFJHW1GYixeL, address:104.36.82.15:4260): protocol: rpc: dial tcp 104.36.82.15:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.213108492 (27h10m58.994041862s) [noise=true , address=104.36.82.55:4195 , piece_id=SXMBKXCIEPCXU3SHT4LA2E42KPKN7IWPUO2IBEBUCH5FNUE2MC3Q, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: rpc: dial tcp 104.36.82.55:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.213171807 (27h10m58.994105178s) [noise=true , address=104.36.82.6:4897 , piece_id=2IXCA2BZX2QOE6NBQ33GC7QLWQPNHHODS5N5Q3ULKP5EQPUCSYMQ, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:12qtpRnAcGiHmdR6J6ysdwaLPvejnCvpKm6MggqsezhbZ14Z5Bt, address:104.36.82.6:4897): protocol: rpc: dial tcp 104.36.82.6:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.213237902 (27h10m58.99417127s) [noise=true , address=104.36.82.20:4448 , piece_id=KFHL4CCPJC6KHDDKOEUTRAQ26HBE3FI5S2LWQ5BBGFC3JTP2GLOA, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:12vbYQVrjLwiHTMGsAGAcjze8KeVQUqemqGiVTWF63Xq6F7ABUp, address:104.36.82.20:4448): protocol: rpc: dial tcp 104.36.82.20:4448: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.213252584 (27h10m58.994185957s) [noise=true , address=104.36.82.3:4788 , piece_id=D5FVNP46P4QHT2NOCENVEVFZIIZKUZFT3P2XXF4XKEN3RAAZEGMA, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:127EfVqSahKLLy8nF95kPd1a39xfThK4AKJ2X2XteKVCNntcMD4, address:104.36.82.3:4788): protocol: rpc: dial tcp 104.36.82.3:4788: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.213282152 (27h10m58.994215522s) [noise=true , address=104.36.82.12:4180 , piece_id=RZ7QVLFCNF32SRFIVS2H2EZPXO7BWACSINZ65OF3IUCYDA3JO3OA, seg_pos={0 0}, upload=25916]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:4180): protocol: write tcp 159.223.143.39:58088->104.36.82.12:4180: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:02:43.213316471 (27h10m58.994249852s) [seg_pos={0 0}, upload=25916]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:02:43.239259691 (27h10m59.020193086s) [upload=25916]: Done uploading segments... Jul 28 01:02:43.239291382 (27h10m59.020224819s) [upload=25916]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.030694622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.......X...........X...................X....XX........X..X.] duration: 670.096587ms FAIL (2025-07-28 01:02:21 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 670.160619ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX.......X...........X...................X....XX........X..X] duration: 30.0578275s FAIL (2025-07-28 01:02:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.057874251s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.............XX...........X...X...X......................] duration: 663.189852ms FAIL (2025-07-28 01:02:05 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 663.212559ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [....XXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030902979s FAIL (2025-07-28 01:01:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 12:4163 , piece_id=NWZ6OBD4PFAUUJR4WVB4F24URXPCVLIQHN3KJNKGO4DRR5TK6MFA, seg_pos={0 0}, upload=25901]: Uploading piece... Jul 28 01:00:43.197427059 (27h8m58.978360435s) [noise=true , address=104.36.82.20:4441 , piece_id=EDKJB7XYJZIFWX33TDYZVJSEIWCHXAE3UHZQCGT4LQXS5DKVKZRQ, seg_pos={0 0}, upload=25901]: Uploading piece... Jul 28 01:00:43.197763076 (27h8m58.97869646s) [noise=true , address=79.127.201.215:4350 , piece_id=KITZO46NJH7MR63QEDT6OZNJRIOCYNC32FXOLRVL5OBXISGCFAVA, seg_pos={0 0}, upload=25901]: Uploading piece... Jul 28 01:00:43.198092856 (27h8m58.979026241s) [noise=true , address=104.36.82.41:4791 , piece_id=LKMRQ2RSEXQRC4NM5CP5EXVOMBY2LICKYK627VYSFGV3VDHSO45Q, seg_pos={0 0}, upload=25901]: Uploading piece... Jul 28 01:00:43.198408903 (27h8m58.979342282s) [noise=true , address=104.36.82.52:4110 , piece_id=UVWJS5M62YFXI5ABTYMO7HZSYQAJBY3L22BDPAV6VZTKWXEYFS4Q, seg_pos={0 0}, upload=25901]: Uploading piece... Jul 28 01:00:43.199662967 (27h8m58.980596345s) [noise=true , address=104.36.82.56:4243 , piece_id=FFA3VOC72YQP37YL4A54JSKGDVEU2EFSJLBVRBZ5GZDNLJOP6R7A, seg_pos={0 0}, upload=25901]: Uploading piece... Jul 28 01:00:43.200036350 (27h8m58.980969718s) [noise=true , address=104.36.82.44:4887 , piece_id=4DRH6AIYUKC2XYOTH6MULOYAFT7DODDTPCGKLEFUNCTVFBIUBJEQ, seg_pos={0 0}, upload=25901]: Uploading piece... Jul 28 01:00:43.200655400 (27h8m58.98158878s) [noise=true , address=104.36.82.47:4549 , piece_id=GE6IM4VMBYHV2VEIU6SX6RDA5T3UWR6MFNL2F5WOJMMV7NOHA5JQ, seg_pos={0 0}, upload=25901]: Uploading piece... Jul 28 01:00:43.425452981 (27h8m59.20638638s) [upload=25901]: Next returned nil segment Jul 28 01:00:43.425474728 (27h8m59.206408116s) [upload=25901]: Waiting for error group managing segments... Jul 28 01:00:45.457638093 (27h9m1.238571454s) [noise=true , address=79.127.201.215:4350 , piece_id=KITZO46NJH7MR63QEDT6OZNJRIOCYNC32FXOLRVL5OBXISGCFAVA, seg_pos={0 0}, upload=25901]: Done uploading piece. err: <nil> Jul 28 01:01:43.110594421 (27h9m58.891527789s) [noise=true , address=104.36.82.10:4094 , piece_id=OZGLNL3FVIDDVZYRVVVENSKIPCH6V2SS2NYKJKFSRJG52TRZ2KRA, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:124zBCfZBTW69BSaGRzUyjWUBjSJhaLMZXQYnqMPGSKL4EX6HVV, address:104.36.82.10:4094): protocol: rpc: dial tcp 104.36.82.10:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.110746712 (27h9m58.891680079s) [noise=true , address=104.36.82.52:4110 , piece_id=UVWJS5M62YFXI5ABTYMO7HZSYQAJBY3L22BDPAV6VZTKWXEYFS4Q, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.110813875 (27h9m58.89174725s) [noise=true , address=104.36.82.47:4549 , piece_id=GE6IM4VMBYHV2VEIU6SX6RDA5T3UWR6MFNL2F5WOJMMV7NOHA5JQ, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:1HWoyMKQ6v2XWUfEmfFh3m8AiGivAcYkkZ9na8kqHFzH2hPDeK, address:104.36.82.47:4549): protocol: rpc: dial tcp 104.36.82.47:4549: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.110848205 (27h9m58.891781582s) [noise=true , address=104.36.82.15:4270 , piece_id=JE3PAHISXRKO2UDASOIP5TYOZSP2HRAQ6U64ZGEBUBQTZE77MTFA, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB, address:104.36.82.15:4270): protocol: rpc: dial tcp 104.36.82.15:4270: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.110876628 (27h9m58.891809995s) [noise=true , address=104.36.82.2:4744 , piece_id=QALMLEXAS4W4Y2HXZQWUPQULALPQRBAV75USI4YYGPO5FE4THVDQ, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH, address:104.36.82.2:4744): protocol: rpc: dial tcp 104.36.82.2:4744: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.110936453 (27h9m58.891869819s) [noise=true , address=104.36.82.11:4126 , piece_id=MG2HSWO3C2U6EREEFORUPCPHENVAAM4WAFH3743MNJB4C7RWR2WA, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:12R3SVXrhJsmYTXR4vDSkfaiqDRox21u17SLmQc9sEab1B7zZRr, address:104.36.82.11:4126): protocol: rpc: dial tcp 104.36.82.11:4126: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.110950900 (27h9m58.891884269s) [noise=true , address=104.36.82.20:4441 , piece_id=EDKJB7XYJZIFWX33TDYZVJSEIWCHXAE3UHZQCGT4LQXS5DKVKZRQ, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:104.36.82.20:4441): protocol: write tcp 159.223.143.39:43620->104.36.82.20:4441: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.110997956 (27h9m58.891931333s) [noise=true , address=104.36.82.13:4218 , piece_id=UKPYHSYH67PVOXAOX2R4Z6L76IYQOMM4H4FTT23OU5ZTYMDQJN5Q, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc, address:104.36.82.13:4218): protocol: rpc: dial tcp 104.36.82.13:4218: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.111115902 (27h9m58.892049268s) [noise=true , address=104.36.82.12:4163 , piece_id=NWZ6OBD4PFAUUJR4WVB4F24URXPCVLIQHN3KJNKGO4DRR5TK6MFA, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC, address:104.36.82.12:4163): protocol: rpc: dial tcp 104.36.82.12:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.111137441 (27h9m58.89207082s) [noise=true , address=104.36.82.55:4193 , piece_id=D6NGNELHTYQQMKHX5J45THV77HQDZAVTYWTYOHCLKM7X3RGM4UCQ, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:104.36.82.55:4193): protocol: rpc: dial tcp 104.36.82.55:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.111212725 (27h9m58.892146102s) [noise=true , address=104.36.82.56:4243 , piece_id=FFA3VOC72YQP37YL4A54JSKGDVEU2EFSJLBVRBZ5GZDNLJOP6R7A, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:12Fyogwa9NcRSUwxTPFTGL68Dbn1FEAqs8K48CNtTyKQEgVwY1c, address:104.36.82.56:4243): protocol: rpc: dial tcp 104.36.82.56:4243: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.111223913 (27h9m58.892157275s) [noise=true , address=104.36.82.41:4791 , piece_id=LKMRQ2RSEXQRC4NM5CP5EXVOMBY2LICKYK627VYSFGV3VDHSO45Q, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:12LpFW1mfotWPqgQxyBSimChkttujx21L6rU2GRSwvqbZ7CVeLn, address:104.36.82.41:4791): protocol: rpc: dial tcp 104.36.82.41:4791: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.111306260 (27h9m58.892239628s) [noise=true , address=104.36.82.44:4887 , piece_id=4DRH6AIYUKC2XYOTH6MULOYAFT7DODDTPCGKLEFUNCTVFBIUBJEQ, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H, address:104.36.82.44:4887): protocol: write tcp 159.223.143.39:45162->104.36.82.44:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.111321004 (27h9m58.892254384s) [noise=true , address=104.36.82.14:4228 , piece_id=2LG7HMI7VQLFYWSGJS2UVJACLZEBM3AOHVUNDM7HHGIQ6HWKLGOQ, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 159.223.143.39:58632->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.111384815 (27h9m58.892318195s) [noise=true , address=104.36.82.4:4836 , piece_id=ECYHGF6ABR7NC2SH2OEWR2HLAJOKVHEHBTSUI3V5NAH3DSCXINVA, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:104.36.82.4:4836): protocol: rpc: dial tcp 104.36.82.4:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.111401727 (27h9m58.892335093s) [noise=true , address=104.36.82.47:4537 , piece_id=CLAKYFYXC3R4KBVVNBP2MNGLJMT2P3FFFWQSOBW52RLW3JHDEFNA, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: rpc: dial tcp 104.36.82.47:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.111451884 (27h9m58.892385249s) [noise=true , address=104.36.82.42:4798 , piece_id=Y4PSGBJBZBIWH55QOZHUXNTC7CM7233AEMDFLSB6HRHCG4HWRQMQ, seg_pos={0 0}, upload=25901]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 159.223.143.39:40134->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:01:43.111504510 (27h9m58.892437889s) [seg_pos={0 0}, upload=25901]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:01:43.139238494 (27h9m58.920171873s) [upload=25901]: Done uploading segments... Jul 28 01:01:43.139279530 (27h9m58.920212917s) [upload=25901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.030925895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.............XX...........X...X...X.....................] duration: 762.93195ms FAIL (2025-07-28 01:01:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 0}, upload=25651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 159.223.143.39:39890->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:18.975782256 (26h52m34.756715638s) [seg_pos={0 0}, upload=25651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:44:18.975871874 (26h52m34.756805277s) [upload=25651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:44:18.975878645 (26h52m34.756812034s) [upload=25651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:44:19.019856339 (26h52m34.800789708s) [upload=25651]: Done uploading segments... Jul 28 00:44:19.019879905 (26h52m34.800813283s) [upload=25651]: Upload finished. err: <nil> Jul 28 00:45:02.257192553 (26h53m18.03812594s) [upload=25659]: Starting upload Jul 28 00:45:02.257306836 (26h53m18.038240216s) [upload=25659]: Done starting upload Jul 28 00:45:02.257434360 (26h53m18.038367739s) [upload=25659]: Uploading segments... Jul 28 00:45:02.257858888 (26h53m18.038792251s) [upload=25659]: Got next segment. Inline: false Jul 28 00:45:02.257951856 (26h53m18.038885235s) [upload=25659]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:45:02.257959915 (26h53m18.038893293s) [upload=25659]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:45:02.293048419 (26h53m18.073981806s) [seg_pos={0 0}, upload=25659]: Begin upload segment... Jul 28 00:45:02.293125122 (26h53m18.074058491s) [seg_pos={0 0}, upload=25659]: Done begin upload segment. Jul 28 00:45:02.293171689 (26h53m18.074105073s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293724212 (26h53m18.074657618s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.293744577 (26h53m18.074677964s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294292781 (26h53m18.075226212s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294343706 (26h53m18.075277094s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294756878 (26h53m18.075690269s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.294891918 (26h53m18.075825306s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295221121 (26h53m18.076154513s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295558583 (26h53m18.076491975s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295604310 (26h53m18.0765377s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.295991730 (26h53m18.07692512s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.296088221 (26h53m18.077021615s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.301695563 (26h53m18.082628952s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302203775 (26h53m18.083137164s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302629001 (26h53m18.083562389s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.302995157 (26h53m18.083928524s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Uploading piece... Jul 28 00:45:02.418208178 (26h53m18.19914155s) [upload=25659]: Next returned nil segment Jul 28 00:45:02.418226832 (26h53m18.19916021s) [upload=25659]: Waiting for error group managing segments... Jul 28 00:45:02.424117363 (26h53m18.205050731s) [noise=true , address=77.90.31.15:4699 , piece_id=474CUNSVTINCCZZCXPBET373D7PKQYHRGN32JEJZYKJ3X6ZJM2HQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.427234519 (26h53m18.208167896s) [noise=true , address=45.91.116.14:4189 , piece_id=DRMCWLOOWG2UFFXZA5XQP5JOJUO5WQOOUVLSAWIVF6YK24XRPTPA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.430114806 (26h53m18.211048185s) [noise=true , address=45.91.116.13:4154 , piece_id=UIIVH5FUZSCBP4ACA2426UFJ4YCEKYAI6O7BPTK7QBQYBOA34IZQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.435624120 (26h53m18.216557505s) [noise=true , address=45.91.116.12:4225 , piece_id=WJY77GRZYSH5WBHPC4IN5IYHXPTI233ZMXRZCJU7YJVP5X5OLE2Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436119655 (26h53m18.217053027s) [noise=true , address=77.90.31.21:4879 , piece_id=XQ2HLQBH7MLG5SYFTXXOAQGW2OQ4ELFWMGNQDIQQXSUNWQZ6RGWA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436405218 (26h53m18.217338582s) [noise=true , address=45.91.116.13:4155 , piece_id=AHCB324J4MEQFDIL446YZCQYXEVJLWNVPWDXDW657MADLMHN25NA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436887494 (26h53m18.217820863s) [noise=true , address=77.90.31.12:4589 , piece_id=HKSIQBOQ27NGBZURHEIYPO7NOV4TJV7WAC5N2XFF5NXG457CFF4Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.436998726 (26h53m18.217932114s) [noise=true , address=77.90.31.19:4841 , piece_id=ZYJMISQVDC24SCPRDG44XBW2KXIVZFDW36M6IICMKKRNTNQVAYSA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437527079 (26h53m18.218460459s) [noise=true , address=45.91.116.10:4083 , piece_id=PPSILB2UAR5L62JYMUHLUFRMVBFIION4ZHKOTXDCU77PJR4OOGVQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.437625274 (26h53m18.218558642s) [noise=true , address=77.90.31.18:4805 , piece_id=B227ERF7AGSHZNPPTDXGEE7QOLCWMWQMIHOQH6MQQLBPD2HZZ7WA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439469614 (26h53m18.220402995s) [noise=true , address=45.91.116.4:4940 , piece_id=YOCQOJXIQK3LRVPERFUJL6U3FP2HUHTOJEUG526LUMUOE2GQSHCQ, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:02.439557143 (26h53m18.220490506s) [noise=true , address=77.90.31.11:4553 , piece_id=FDGGPSIYSQ7LH34HNSFTE6HGZQRHRZKRVMLK7GERP2JAKBOYC3QA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: <nil> Jul 28 00:45:32.221331076 (26h53m48.002264449s) [noise=true , address=104.36.82.16:4295 , piece_id=5RGIRJLSFZSCVHCBV5IPLBKLDY3AJWIHGUYYWPBU4OJHI7JCOV4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:33378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221411781 (26h53m48.002345162s) [noise=true , address=104.36.82.7:4910 , piece_id=RZWYNG5BUBMWZQXM5PORO6OGAX56Q7BKIOM5TG2NYIMOBK5EZBJA, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221457184 (26h53m48.002390547s) [noise=true , address=104.36.82.13:4187 , piece_id=ASDURYYEDLBUWLT3WMGU3X2ZVACPRQEBT6PPOAB34AHTK7R3354Q, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221473904 (26h53m48.002407294s) [noise=true , address=104.36.82.8:4969 , piece_id=GB6IZEVKFMV77V3NA2J2SMJGYOHQDFMVCFGXEQL5UNBVOMBEAY4A, seg_pos={0 0}, upload=25659]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:49546->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:32.221516309 (26h53m48.002449671s) [seg_pos={0 0}, upload=25659]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:45:32.251981717 (26h53m48.032915086s) [upload=25659]: Done uploading segments... Jul 28 00:45:32.252005327 (26h53m48.032938704s) [upload=25659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 762.961304ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XX.......X...........X...................X....XX........X..] duration: 630.938026ms FAIL (2025-07-28 01:01:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 631.159188ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [.....XXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.026362044s FAIL (2025-07-28 01:00:42 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 653787s) [noise=true , address=104.36.82.7:4929 , piece_id=MIQKPMZBERJ4IOXC6TVCYVLU3JFT7KM5AKKYTOARYJJ3HXD43FQQ, seg_pos={0 0}, upload=25885]: Uploading piece... Jul 28 00:59:43.028120983 (27h7m58.809054368s) [noise=true , address=104.36.82.20:4468 , piece_id=SRQKV2NS44HM45D7H77FEZO4FMC7XL6HV2D3Q5RCFEGWSDMFQVEQ, seg_pos={0 0}, upload=25885]: Uploading piece... Jul 28 00:59:43.028356982 (27h7m58.80929037s) [noise=true , address=104.36.82.51:4059 , piece_id=L7DIQ2NEGW6DTVF4ZVTBSF223PP5KI5LA5WPWLJRVASPRZY6TMXQ, seg_pos={0 0}, upload=25885]: Uploading piece... Jul 28 00:59:43.028854705 (27h7m58.80978822s) [noise=true , address=104.36.82.4:4832 , piece_id=SGEEIONDETZXTIMDCAK2M3NCZ4WRWVEJEW5OQ56MRKTSXCDLHGVA, seg_pos={0 0}, upload=25885]: Uploading piece... Jul 28 00:59:43.029305809 (27h7m58.810239198s) [noise=true , address=104.36.82.11:4144 , piece_id=WI55XBQI7WP7VA4HP65SCIW2OB7VI5N2JKNB6UPJDWL4TIE5GYZA, seg_pos={0 0}, upload=25885]: Uploading piece... Jul 28 00:59:43.029720308 (27h7m58.810653695s) [noise=true , address=104.36.82.44:4895 , piece_id=ZN7IHG6X7WS3OP7T5JA6WD5G7PF5EJX4PEFBXD23XF43GIZF3XSA, seg_pos={0 0}, upload=25885]: Uploading piece... Jul 28 00:59:43.030251714 (27h7m58.811185141s) [noise=true , address=104.36.82.52:4098 , piece_id=AN5IKL64ULEFHTY2DZI2FLIHJMPPOZVFI777BEHK46O5GFY5V64Q, seg_pos={0 0}, upload=25885]: Uploading piece... Jul 28 00:59:43.030653256 (27h7m58.811586642s) [noise=true , address=104.36.82.3:4775 , piece_id=FHHGAWGUMYP4VO7ALVGUZ2QKXLILRUBSWUEPJ2DMCPAKYRB26BHQ, seg_pos={0 0}, upload=25885]: Uploading piece... Jul 28 00:59:43.230361119 (27h7m59.011294498s) [upload=25885]: Next returned nil segment Jul 28 00:59:43.230393314 (27h7m59.011326699s) [upload=25885]: Waiting for error group managing segments... Jul 28 00:59:47.895773632 (27h8m3.676707009s) [noise=true , address=79.127.201.215:4346 , piece_id=D7NMWCULAFGB4APQZTKSF4RMRJTTW7W643HIWBX7OT34L3SDZC3Q, seg_pos={0 0}, upload=25885]: Done uploading piece. err: <nil> Jul 28 01:00:42.949774638 (27h8m58.730708007s) [noise=true , address=104.36.82.47:4531 , piece_id=6TWNJBH5HSDL2QRUE5G66OEBP7YA33YY4A5BYB627OUQXA4NJGIQ, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.949843168 (27h8m58.730776533s) [noise=true , address=104.36.82.20:4468 , piece_id=SRQKV2NS44HM45D7H77FEZO4FMC7XL6HV2D3Q5RCFEGWSDMFQVEQ, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:4468): protocol: rpc: dial tcp 104.36.82.20:4468: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.949915516 (27h8m58.730848881s) [noise=true , address=104.36.82.3:4775 , piece_id=FHHGAWGUMYP4VO7ALVGUZ2QKXLILRUBSWUEPJ2DMCPAKYRB26BHQ, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:12BvQLLnX5A2BgKeqn1ubs5iBeCUsUGH8hFjpB2HvY53YH9Nw1n, address:104.36.82.3:4775): protocol: rpc: dial tcp 104.36.82.3:4775: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.949969626 (27h8m58.730903s) [noise=true , address=104.36.82.4:4832 , piece_id=SGEEIONDETZXTIMDCAK2M3NCZ4WRWVEJEW5OQ56MRKTSXCDLHGVA, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:1SWdNMwLYv7KiBqr5Pfo8qvxHnxirHGUDbhZWeojGXwueQUqr9, address:104.36.82.4:4832): protocol: rpc: dial tcp 104.36.82.4:4832: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950015881 (27h8m58.730949257s) [noise=true , address=104.36.82.52:4098 , piece_id=AN5IKL64ULEFHTY2DZI2FLIHJMPPOZVFI777BEHK46O5GFY5V64Q, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:1SoFNdwfh4p5tXUkpfjQaev6tdA2ZdhsXUFvZNBsy5QrJ9zdqb, address:104.36.82.52:4098): protocol: rpc: dial tcp 104.36.82.52:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950064576 (27h8m58.730997945s) [noise=true , address=104.36.82.56:4259 , piece_id=FJTLK5YP2US3MLXCIRGYLI75QJK3KHWQ6WUBKCFJYO5QF4PPXTLQ, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: write tcp 159.223.143.39:33990->104.36.82.56:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950088684 (27h8m58.731022049s) [noise=true , address=104.36.82.7:4929 , piece_id=MIQKPMZBERJ4IOXC6TVCYVLU3JFT7KM5AKKYTOARYJJ3HXD43FQQ, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:12CAQMW1ZMUbDAfcEGTmeeBfWqz3tjbPwz33XvWSAxZLsgts1wT, address:104.36.82.7:4929): protocol: rpc: dial tcp 104.36.82.7:4929: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950112675 (27h8m58.731046041s) [noise=true , address=104.36.82.49:4979 , piece_id=ECOVXE4EG2WNLB3CDYRCAQILHGBLTBATW6SDY2WCZXNZ2OVME5PQ, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950164780 (27h8m58.731098166s) [noise=true , address=104.36.82.51:4059 , piece_id=L7DIQ2NEGW6DTVF4ZVTBSF223PP5KI5LA5WPWLJRVASPRZY6TMXQ, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950226280 (27h8m58.731159656s) [noise=true , address=104.36.82.54:4170 , piece_id=RWFHRDWJ2MI47L6VCS5SCLWKUFGVQYARNCEVGBPZYLHQI35FDYVA, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX, address:104.36.82.54:4170): protocol: write tcp 159.223.143.39:33912->104.36.82.54:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950241584 (27h8m58.731174963s) [noise=true , address=104.36.82.44:4895 , piece_id=ZN7IHG6X7WS3OP7T5JA6WD5G7PF5EJX4PEFBXD23XF43GIZF3XSA, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: rpc: dial tcp 104.36.82.44:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950253604 (27h8m58.731186973s) [noise=true , address=104.36.82.11:4144 , piece_id=WI55XBQI7WP7VA4HP65SCIW2OB7VI5N2JKNB6UPJDWL4TIE5GYZA, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:128246NkJjT3UFiQdSqFssY6jTAV1jye3dXAjAkFnC3DUAXb235, address:104.36.82.11:4144): protocol: rpc: dial tcp 104.36.82.11:4144: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950263450 (27h8m58.731196824s) [noise=true , address=104.36.82.13:4219 , piece_id=NEIQPR556CHSEHJULQUUISL2EZ77K6BF7CLPR6GPGATG6MSKLJ3Q, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:12fUzLg93yKvyKkNvuvcKZHyvJ2HGE6tUCg1Wm7ZZjKKeDZDWm9, address:104.36.82.13:4219): protocol: rpc: dial tcp 104.36.82.13:4219: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950294670 (27h8m58.731228045s) [noise=true , address=104.36.82.10:4083 , piece_id=BO5NCGPUZYVWC2MZD425Z3GA36RTTWQCQNZMWOXUG25R5ZNKTOMQ, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh, address:104.36.82.10:4083): protocol: rpc: dial tcp 104.36.82.10:4083: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950306812 (27h8m58.731240182s) [noise=true , address=104.36.82.50:4021 , piece_id=E5AVCBQYNU5DNVIV4PUVSVDQHZ3ROSWDZNZO5PTN3Q426JAIGIOQ, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:1nz7Gxp4YTWuB1JDnjE5Q9BfVUUvyZ4SMBbZrxVQHrSUvcJbAQ, address:104.36.82.50:4021): protocol: rpc: dial tcp 104.36.82.50:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950319914 (27h8m58.73125329s) [noise=true , address=104.36.82.14:4227 , piece_id=P2JTJKEMTZWRLXO4AFJAJFC7UIQ6UFV25MKDF2ZW4MTZLVL5R6GA, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:104.36.82.14:4227): protocol: rpc: dial tcp 104.36.82.14:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950335526 (27h8m58.731268893s) [noise=true , address=104.36.82.21:4496 , piece_id=CLVKLJNXVWKQ6DWXVOVFTUO4AWYIGV7LZRU35D5NRZLOUMILEMIQ, seg_pos={0 0}, upload=25885]: Done uploading piece. err: ecclient: upload failed (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w, address:104.36.82.21:4496): protocol: write tcp 159.223.143.39:45926->104.36.82.21:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:00:42.950386334 (27h8m58.731319716s) [seg_pos={0 0}, upload=25885]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:00:42.973257567 (27h8m58.754190942s) [upload=25885]: Done uploading segments... Jul 28 01:00:42.973287717 (27h8m58.754221095s) [upload=25885]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.026381872s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)