Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 30.045171549s FAIL (2025-07-26 23:19:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== Jul 26 23:19:02.159755103 (1h27m17.940688492s) [noise=true , address=104.36.82.12:4150 , piece_id=FI3GLMHJP7HZZJX3XXVC2E3GS6IUXDZNLYGSM4MSKH2F64YNMEXA, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.160279772 (1h27m17.941213163s) [noise=true , address=104.36.82.6:4876 , piece_id=KR6TJ2HCFCXTJISM6UPV7JUNSBROCDCKMTOSOFCRDJHPUPSAFMLQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.161927848 (1h27m17.942861233s) [noise=true , address=104.36.82.6:4874 , piece_id=MVSAK5VFA2G3KNI2P6UCDD4LOIURKXS64WG22366TH5ILGIICKMQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.162315475 (1h27m17.943248858s) [noise=true , address=104.36.82.4:4802 , piece_id=NBH4NEELW65F4QNPEK7HQZ54HTSPKZIN4OPFRJVBA2WESXKY6Q3Q, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.162713925 (1h27m17.943647313s) [noise=true , address=104.36.82.10:4077 , piece_id=LFVEQNFIIGJKSSTQXMEI3ZXWFUEISERM44XRRNTR5SS3OAA2HZBQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.163137239 (1h27m17.944070634s) [noise=true , address=104.36.82.3:4767 , piece_id=PYJ32GWWGMZ2M2N3EXHI2CLQIBH7Y7PPGHSOVPU2DSXEDQU3T3AA, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.163597043 (1h27m17.944530432s) [noise=true , address=45.91.116.7:4010 , piece_id=NVW7C7WAESRXQHLRMSUOHN4VGLQ4JUU4ONZDKYA74HRWNPJ77LRA, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.164025355 (1h27m17.944958745s) [noise=true , address=77.90.31.17:4771 , piece_id=4W7HIAT4QT6AI25QFB32ZTE5G7QVBXGEJWFCKQAEXMWODF2BWHHQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.164464623 (1h27m17.945397997s) [noise=true , address=104.36.82.5:4837 , piece_id=4GM4QNJ4NDEWV6X663ZIXKWTNHABY5NVCW4CSE522OF2OIU3LX6Q, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.164960445 (1h27m17.945893835s) [noise=true , address=104.36.82.8:4972 , piece_id=37M6QNQH6ZB5KIRIJF52JGP7XTQVNI3ZCW7OHF2GTGLIMRFMBALQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.165341867 (1h27m17.946275257s) [noise=true , address=45.91.116.10:4082 , piece_id=B4EMTRIN53U6G34AW46P6CE67M6A3JJSVMBYDPI3JKWBN32YYAVA, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.165747422 (1h27m17.94668081s) [noise=true , address=45.91.116.6:4975 , piece_id=TCZUQCESMM65HGJH3ARHRLVDCCWK2P3LUQJPPRJ2SCPVHOCLWJMQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.166129531 (1h27m17.947062921s) [noise=true , address=77.90.31.10:4518 , piece_id=MSBETVWMIRSWRNMIP73TA3MPBSNMTUBEOI2GG4JNTROCJDBE5SKA, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.178017909 (1h27m17.9589513s) [noise=true , address=104.36.82.13:4185 , piece_id=TVHBKI2EHIGUTRHZ2YBA4B75KMAP46TUSCJYSKLGQIHOZWJLRN4A, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.274565238 (1h27m18.055498628s) [upload=1346]: Next returned nil segment Jul 26 23:19:02.274595985 (1h27m18.055529374s) [upload=1346]: Waiting for error group managing segments... Jul 26 23:19:02.280814802 (1h27m18.061748183s) [noise=true , address=45.91.116.7:4010 , piece_id=NVW7C7WAESRXQHLRMSUOHN4VGLQ4JUU4ONZDKYA74HRWNPJ77LRA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: <nil> Jul 26 23:19:02.281542924 (1h27m18.062476311s) [noise=true , address=45.91.116.10:4082 , piece_id=B4EMTRIN53U6G34AW46P6CE67M6A3JJSVMBYDPI3JKWBN32YYAVA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: <nil> Jul 26 23:19:02.283976464 (1h27m18.064909847s) [noise=true , address=45.91.116.6:4975 , piece_id=TCZUQCESMM65HGJH3ARHRLVDCCWK2P3LUQJPPRJ2SCPVHOCLWJMQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: <nil> Jul 26 23:19:32.077949022 (1h27m47.858882402s) [noise=true , address=104.36.82.9:4007 , piece_id=XNHF2LWRAOPCMRYWTDCAAGMTHKF5NWGJHEDIH2AXJKBZTUHA6HKA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:50642->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078061043 (1h27m47.858994464s) [noise=true , address=104.36.82.10:4077 , piece_id=LFVEQNFIIGJKSSTQXMEI3ZXWFUEISERM44XRRNTR5SS3OAA2HZBQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:58876->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078180247 (1h27m47.859113627s) [noise=true , address=104.36.82.6:4876 , piece_id=KR6TJ2HCFCXTJISM6UPV7JUNSBROCDCKMTOSOFCRDJHPUPSAFMLQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:51466->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078265362 (1h27m47.85919875s) [noise=true , address=104.36.82.13:4185 , piece_id=TVHBKI2EHIGUTRHZ2YBA4B75KMAP46TUSCJYSKLGQIHOZWJLRN4A, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:53170->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078339117 (1h27m47.859272487s) [noise=true , address=104.36.82.7:4909 , piece_id=P34DSKK7D6BLWPRW5YYUHFX2H4OG55JJUEBGWO2Y4SAT6NR7GZRQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 159.223.143.39:38784->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078634690 (1h27m47.859568074s) [noise=true , address=104.36.82.6:4874 , piece_id=MVSAK5VFA2G3KNI2P6UCDD4LOIURKXS64WG22366TH5ILGIICKMQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078667658 (1h27m47.859601041s) [noise=true , address=104.36.82.3:4767 , piece_id=PYJ32GWWGMZ2M2N3EXHI2CLQIBH7Y7PPGHSOVPU2DSXEDQU3T3AA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078763314 (1h27m47.859696696s) [noise=true , address=104.36.82.4:4802 , piece_id=NBH4NEELW65F4QNPEK7HQZ54HTSPKZIN4OPFRJVBA2WESXKY6Q3Q, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078833065 (1h27m47.859766446s) [noise=true , address=104.36.82.8:4972 , piece_id=37M6QNQH6ZB5KIRIJF52JGP7XTQVNI3ZCW7OHF2GTGLIMRFMBALQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078870910 (1h27m47.859804295s) [noise=true , address=77.90.31.17:4771 , piece_id=4W7HIAT4QT6AI25QFB32ZTE5G7QVBXGEJWFCKQAEXMWODF2BWHHQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078905195 (1h27m47.859838558s) [noise=true , address=104.36.82.12:4150 , piece_id=FI3GLMHJP7HZZJX3XXVC2E3GS6IUXDZNLYGSM4MSKH2F64YNMEXA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55474->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078929978 (1h27m47.85986335s) [noise=true , address=104.36.82.5:4837 , piece_id=4GM4QNJ4NDEWV6X663ZIXKWTNHABY5NVCW4CSE522OF2OIU3LX6Q, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078963908 (1h27m47.859897293s) [noise=true , address=77.90.31.10:4518 , piece_id=MSBETVWMIRSWRNMIP73TA3MPBSNMTUBEOI2GG4JNTROCJDBE5SKA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.079013476 (1h27m47.859946866s) [seg_pos={0 0}, upload=1346]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Jul 26 23:19:32.118793433 (1h27m47.899726822s) [upload=1346]: Done uploading segments... Jul 26 23:19:32.118867751 (1h27m47.899801128s) [upload=1346]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 30.045197889s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXX] duration: 1m0.030011889s FAIL (2025-07-26 23:19:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 18:17.971724315 (1h26m33.752657684s) [upload=1336]: Uploading segments... Jul 26 23:18:17.972298458 (1h26m33.753231828s) [upload=1336]: Got next segment. Inline: false Jul 26 23:18:17.972377867 (1h26m33.753311231s) [upload=1336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:18:17.972384372 (1h26m33.75331775s) [upload=1336]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:18:18.011255720 (1h26m33.792189092s) [seg_pos={0 0}, upload=1336]: Begin upload segment... Jul 26 23:18:18.011308406 (1h26m33.792241772s) [seg_pos={0 0}, upload=1336]: Done begin upload segment. Jul 26 23:18:18.011351967 (1h26m33.792285334s) [noise=true , address=79.127.205.233:4653 , piece_id=SB7L2LSSQKE3XTW7IKQQYJI4KRFDFRVQZV7LXEOCKZQCKGUSU6OQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.011811757 (1h26m33.792745139s) [noise=true , address=79.127.205.245:4225 , piece_id=MDPVVKF3JJJEJJRAP645ZBVV2LKDDIGOF2L4WJBNRTAMLT7NAJEA, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.012328138 (1h26m33.79326152s) [noise=true , address=104.36.82.54:4159 , piece_id=A3OBET2XDW7WKMLL4XCVQZ5WBVE633MY3U72LRCIWLFISREGPTZQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.012868297 (1h26m33.793801675s) [noise=true , address=104.36.82.53:4129 , piece_id=2J3SPS2TCSSKWIV22J2DWMF65LNFIU4FPKU7VMSV3RCADJWBYMSQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.013183201 (1h26m33.794116583s) [noise=true , address=77.90.31.13:4638 , piece_id=CEBNAK2FPBS44DMBK4MVYYBTYNIYCJKBR6L3X3R2QXJPWQ7SWKMA, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.013544174 (1h26m33.794477554s) [noise=true , address=79.127.201.215:4357 , piece_id=UAG3CLZA44VTWYFB4B74CG7I4U6JQFIEDAJBWC4AVROWYN7Y7W4Q, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.013877533 (1h26m33.794810915s) [noise=true , address=45.91.116.11:4151 , piece_id=37N7LUHOAW4PHDVEFRNIEQX4376CF3WEVVXINEE7CZKYLAMMN4RQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.014191229 (1h26m33.795124611s) [noise=true , address=104.36.82.46:4506 , piece_id=RROKRSO5GHA6NKRC7ZAIV6K4BUSIQCCA6XMEDC26WITGRFI35ANA, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.014598985 (1h26m33.795532368s) [noise=true , address=77.90.31.21:4902 , piece_id=WEJPL4MORPE3WIQC44QD4KB2H3YU7TWW2MYBPBLX4Y2IZBAZI27Q, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.017833696 (1h26m33.798767084s) [noise=true , address=45.91.116.14:4194 , piece_id=SK2WCNUOHLXCLQAXTYZYRTUWVA6AYQPN5UPQDZ2CID56XMP4H6XQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.018771406 (1h26m33.799704796s) [noise=true , address=104.36.82.47:4517 , piece_id=Y32KIPCALDESHPCDZTC5SAILDRHZH427OPTNWSA2J5UJ3BLRXGRQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.018907007 (1h26m33.7998404s) [noise=true , address=64.49.13.139:4478 , piece_id=PPZFKLQNUM2SUQX2MLYPFQLDVZ72WANLKJSD5STWOI7DFD7SOOIA, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.019355074 (1h26m33.800288457s) [noise=true , address=64.49.13.138:4483 , piece_id=LV72Z4XG6PDACBVTNRURVEDOHF6DTP6LUD6MF3K2IJ5CE7GY7M2Q, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.019718524 (1h26m33.800651907s) [noise=true , address=104.36.82.52:4099 , piece_id=U2JHRJRLRSFE5JN25LTDJFWCRI4CGL2BJ7CFVTPXZ5DQFLLNN7QQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.020086610 (1h26m33.801019997s) [noise=true , address=77.90.31.19:4860 , piece_id=2DABZ3ERA6BSOSKLDCZRDJJWFE5KULV4GR5EBFVO72O247DGPX3A, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.020448480 (1h26m33.801381863s) [noise=true , address=45.91.116.13:4175 , piece_id=LXPOY3TJAGAGIU2PJSRSXOG6YSAT36HFEBNLJUQHS3557TYNDFHQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.020778809 (1h26m33.801712195s) [noise=true , address=104.36.82.10:4096 , piece_id=7K4DJTT2SWSEDTT3ZOM5A4KM7OAOS4MFYLYM65X4AME22M2RCB5A, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.021340286 (1h26m33.802273684s) [noise=true , address=79.127.219.38:4004 , piece_id=RONGZNUD64KQITKQR6QVAWMWRIH4LJFBBXXCBBJD5RLROFBNZ7VA, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.134472270 (1h26m33.915405653s) [upload=1336]: Next returned nil segment Jul 26 23:18:18.134498115 (1h26m33.915431502s) [upload=1336]: Waiting for error group managing segments... Jul 26 23:18:18.161612774 (1h26m33.94254615s) [noise=true , address=45.91.116.11:4151 , piece_id=37N7LUHOAW4PHDVEFRNIEQX4376CF3WEVVXINEE7CZKYLAMMN4RQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.161709861 (1h26m33.94264323s) [noise=true , address=104.36.82.54:4159 , piece_id=A3OBET2XDW7WKMLL4XCVQZ5WBVE633MY3U72LRCIWLFISREGPTZQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.192644699 (1h26m33.973578081s) [noise=true , address=77.90.31.21:4902 , piece_id=WEJPL4MORPE3WIQC44QD4KB2H3YU7TWW2MYBPBLX4Y2IZBAZI27Q, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.309825368 (1h26m34.090758733s) [noise=true , address=79.127.219.38:4004 , piece_id=RONGZNUD64KQITKQR6QVAWMWRIH4LJFBBXXCBBJD5RLROFBNZ7VA, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.635791122 (1h26m34.416724508s) [noise=true , address=79.127.205.233:4653 , piece_id=SB7L2LSSQKE3XTW7IKQQYJI4KRFDFRVQZV7LXEOCKZQCKGUSU6OQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.704468976 (1h26m34.485402345s) [noise=true , address=79.127.205.245:4225 , piece_id=MDPVVKF3JJJEJJRAP645ZBVV2LKDDIGOF2L4WJBNRTAMLT7NAJEA, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.809846319 (1h26m34.590779707s) [noise=true , address=77.90.31.13:4638 , piece_id=CEBNAK2FPBS44DMBK4MVYYBTYNIYCJKBR6L3X3R2QXJPWQ7SWKMA, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:23.139703097 (1h26m38.920636463s) [noise=true , address=64.49.13.138:4483 , piece_id=LV72Z4XG6PDACBVTNRURVEDOHF6DTP6LUD6MF3K2IJ5CE7GY7M2Q, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:23.220133506 (1h26m39.001066873s) [noise=true , address=45.91.116.14:4194 , piece_id=SK2WCNUOHLXCLQAXTYZYRTUWVA6AYQPN5UPQDZ2CID56XMP4H6XQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:24.564763574 (1h26m40.345696941s) [noise=true , address=64.49.13.139:4478 , piece_id=PPZFKLQNUM2SUQX2MLYPFQLDVZ72WANLKJSD5STWOI7DFD7SOOIA, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:28.650436015 (1h26m44.431369394s) [noise=true , address=45.91.116.13:4175 , piece_id=LXPOY3TJAGAGIU2PJSRSXOG6YSAT36HFEBNLJUQHS3557TYNDFHQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:33.004606556 (1h26m48.785539941s) [noise=true , address=77.90.31.19:4860 , piece_id=2DABZ3ERA6BSOSKLDCZRDJJWFE5KULV4GR5EBFVO72O247DGPX3A, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:51.458187265 (1h27m7.239120627s) [noise=true , address=79.127.201.215:4357 , piece_id=UAG3CLZA44VTWYFB4B74CG7I4U6JQFIEDAJBWC4AVROWYN7Y7W4Q, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:19:17.942913236 (1h27m33.723846606s) [noise=true , address=104.36.82.47:4517 , piece_id=Y32KIPCALDESHPCDZTC5SAILDRHZH427OPTNWSA2J5UJ3BLRXGRQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: rpc: dial tcp 104.36.82.47:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:17.943006298 (1h27m33.723939669s) [noise=true , address=104.36.82.52:4099 , piece_id=U2JHRJRLRSFE5JN25LTDJFWCRI4CGL2BJ7CFVTPXZ5DQFLLNN7QQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: ecclient: upload failed (node:122dGFWNmsuheRmfEfNGWHftPpCcWv7HHXKymgJxsye6P7ZKhZT, address:104.36.82.52:4099): protocol: write tcp 159.223.143.39:49052->104.36.82.52:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:17.943097246 (1h27m33.724030632s) [noise=true , address=104.36.82.53:4129 , piece_id=2J3SPS2TCSSKWIV22J2DWMF65LNFIU4FPKU7VMSV3RCADJWBYMSQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: write tcp 159.223.143.39:35032->104.36.82.53:4129: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:17.943192577 (1h27m33.724125957s) [noise=true , address=104.36.82.10:4096 , piece_id=7K4DJTT2SWSEDTT3ZOM5A4KM7OAOS4MFYLYM65X4AME22M2RCB5A, seg_pos={0 0}, upload=1336]: Done uploading piece. err: ecclient: upload failed (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA, address:104.36.82.10:4096): protocol: write tcp 159.223.143.39:44950->104.36.82.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:17.943242578 (1h27m33.724175958s) [noise=true , address=104.36.82.46:4506 , piece_id=RROKRSO5GHA6NKRC7ZAIV6K4BUSIQCCA6XMEDC26WITGRFI35ANA, seg_pos={0 0}, upload=1336]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: write tcp 159.223.143.39:49472->104.36.82.46:4506: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:17.943269366 (1h27m33.724202753s) [seg_pos={0 0}, upload=1336]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:19:17.970217453 (1h27m33.751150847s) [upload=1336]: Done uploading segments... Jul 26 23:19:17.970248140 (1h27m33.751181525s) [upload=1336]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.030053831s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............................................XXX.X....XX.X.X] duration: 30.001644783s FAIL (2025-07-26 23:19:00 UTC) orbiter warning: context deadline exceeded Duration: 30.001668119s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 17 out of 60 is failed (threshold=3) [.............X........................X......XXXXXXXXXXXXXXX] duration: 30.03266378s FAIL (2025-07-26 23:18:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.032701707s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.XX.................X.X..........XX...X...X..XX.X...X..X....] duration: 33.242191064s FAIL (2025-07-26 23:18:44 UTC) context deadline exceeded Duration: 33.242207357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 30.049722696s FAIL (2025-07-26 23:18:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 525296s) [noise=true , address=45.91.116.4:4938 , piece_id=63DPNTD3GFXJ7IY3OKRGMIC4WGZ4D3EVKP4V3KQFOKZOGDWBML3A, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.279545133 (1h26m16.060478523s) [noise=true , address=104.36.82.2:4729 , piece_id=G6QQVY57BUVXXXH7Y724JVTBLLRBCVAN7MACG23UZ2Z57FJVWFYQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.280789217 (1h26m16.061722607s) [noise=true , address=45.91.116.14:4191 , piece_id=X6RMIDVE5G46AU2MTXCXELVCKHDFJPPYUIMU5L62DFBFOC5W75AA, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.282241259 (1h26m16.06317464s) [noise=true , address=104.36.82.8:4971 , piece_id=FLVKBFJQNVZTW2B3WRDMGKVPNNNJEY7YTGKLENSDR6QJER2JE32A, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.292222168 (1h26m16.073155587s) [noise=true , address=104.36.82.4:4802 , piece_id=DS37Z4SGBS6GAPJHE6QQJRNYG2FLO5JVW556OXVRCSPGKDZU4IPQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.301089564 (1h26m16.082022953s) [noise=true , address=45.91.116.12:4226 , piece_id=XAUCDHZ54UH52REUL5YSJRP4EPAZAWUJMGA4N5RYJQYXAUV2BWVQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.305127147 (1h26m16.086060537s) [noise=true , address=104.36.82.2:4731 , piece_id=PQWR56TKNZFB6KC5UBKIG5RIJMTHHUZOFPHJC5AUEO7N4SRRPTVA, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.308462552 (1h26m16.089395938s) [noise=true , address=45.91.116.7:4011 , piece_id=67NFFWCGIZS4XE5FZ7G4VYLHOSULQNNR7QSSUZCPGRDWHBBLOYFQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.309207438 (1h26m16.090140828s) [noise=true , address=104.36.82.5:4838 , piece_id=ERVCAMGCRELCFK4N3Z7XDTI57O7YDIL2DYLJIZW6JE3CNVNY3IJQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.310172279 (1h26m16.09110567s) [noise=true , address=104.36.82.10:4077 , piece_id=CSLHKKTCR3E65ST27D4QNAIO5HAJFO2EBR6JGJHSGG2YWER2R3CQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.314176624 (1h26m16.095110011s) [noise=true , address=104.36.82.7:4910 , piece_id=CBHM774RCIRDR75WJEAA4WPZ7GKL4LDLGUUD6MIB5PAPPK5MIVSQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.321471285 (1h26m16.102404672s) [noise=true , address=104.36.82.13:4187 , piece_id=QUPS4ZIJC5HVFO2E3IVNLYXKVIR5HOYR2I2MNFZF7OO4DUGMQGFA, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.327408186 (1h26m16.108341565s) [noise=true , address=104.36.82.15:4259 , piece_id=EO6AL7XRVOEK4Q2UG4YRG66CTLF2J6LGQWT3BSEE4ZIBMQ3AQ4MQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.332643861 (1h26m16.11357725s) [noise=true , address=104.36.82.11:4115 , piece_id=W6QKDCJOUTT63R62I2PH7CFIF5REM4ABYHLBLDMOVMSWEDKINJLQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.351499096 (1h26m16.132432486s) [noise=true , address=77.90.31.10:4518 , piece_id=AZWHPHYYSW4DTYG2E4KY2FH5375VC7EXSQIZ5N6ZIN3D3FRTEUBA, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.355740739 (1h26m16.136674126s) [noise=true , address=104.36.82.7:4909 , piece_id=X2TICLLMHNWCH5ZD6NYOSXPPNHVJF5RFAOSUTDERWENJTSIONLBA, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.435795809 (1h26m16.216729186s) [upload=1330]: Next returned nil segment Jul 26 23:18:00.435818234 (1h26m16.21675162s) [upload=1330]: Waiting for error group managing segments... Jul 26 23:18:00.441544843 (1h26m16.222478211s) [noise=true , address=45.91.116.7:4011 , piece_id=67NFFWCGIZS4XE5FZ7G4VYLHOSULQNNR7QSSUZCPGRDWHBBLOYFQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: <nil> Jul 26 23:18:00.441626722 (1h26m16.222560101s) [noise=true , address=45.91.116.12:4226 , piece_id=XAUCDHZ54UH52REUL5YSJRP4EPAZAWUJMGA4N5RYJQYXAUV2BWVQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: <nil> Jul 26 23:18:00.442183212 (1h26m16.223116578s) [noise=true , address=45.91.116.14:4191 , piece_id=X6RMIDVE5G46AU2MTXCXELVCKHDFJPPYUIMU5L62DFBFOC5W75AA, seg_pos={0 0}, upload=1330]: Done uploading piece. err: <nil> Jul 26 23:18:00.443858435 (1h26m16.224791831s) [noise=true , address=45.91.116.4:4938 , piece_id=63DPNTD3GFXJ7IY3OKRGMIC4WGZ4D3EVKP4V3KQFOKZOGDWBML3A, seg_pos={0 0}, upload=1330]: Done uploading piece. err: <nil> Jul 26 23:18:07.722459075 (1h26m23.503392453s) [noise=true , address=104.36.82.13:4187 , piece_id=QUPS4ZIJC5HVFO2E3IVNLYXKVIR5HOYR2I2MNFZF7OO4DUGMQGFA, seg_pos={0 0}, upload=1330]: Done uploading piece. err: <nil> Jul 26 23:18:30.211825345 (1h26m45.992758724s) [noise=true , address=104.36.82.10:4077 , piece_id=CSLHKKTCR3E65ST27D4QNAIO5HAJFO2EBR6JGJHSGG2YWER2R3CQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:43120->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212493127 (1h26m45.993426492s) [noise=true , address=104.36.82.2:4729 , piece_id=G6QQVY57BUVXXXH7Y724JVTBLLRBCVAN7MACG23UZ2Z57FJVWFYQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212574327 (1h26m45.993507808s) [noise=true , address=104.36.82.7:4910 , piece_id=CBHM774RCIRDR75WJEAA4WPZ7GKL4LDLGUUD6MIB5PAPPK5MIVSQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 159.223.143.39:50510->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212653579 (1h26m45.99358696s) [noise=true , address=77.90.31.10:4518 , piece_id=AZWHPHYYSW4DTYG2E4KY2FH5375VC7EXSQIZ5N6ZIN3D3FRTEUBA, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 159.223.143.39:34986->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212740536 (1h26m45.993673914s) [noise=true , address=104.36.82.7:4909 , piece_id=X2TICLLMHNWCH5ZD6NYOSXPPNHVJF5RFAOSUTDERWENJTSIONLBA, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 159.223.143.39:58768->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212816157 (1h26m45.993749525s) [noise=true , address=104.36.82.8:4971 , piece_id=FLVKBFJQNVZTW2B3WRDMGKVPNNNJEY7YTGKLENSDR6QJER2JE32A, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:43036->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212884145 (1h26m45.993817516s) [noise=true , address=104.36.82.4:4802 , piece_id=DS37Z4SGBS6GAPJHE6QQJRNYG2FLO5JVW556OXVRCSPGKDZU4IPQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212946298 (1h26m45.993879676s) [noise=true , address=104.36.82.15:4259 , piece_id=EO6AL7XRVOEK4Q2UG4YRG66CTLF2J6LGQWT3BSEE4ZIBMQ3AQ4MQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.213025924 (1h26m45.993959292s) [noise=true , address=104.36.82.11:4115 , piece_id=W6QKDCJOUTT63R62I2PH7CFIF5REM4ABYHLBLDMOVMSWEDKINJLQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:37348->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.213077537 (1h26m45.994010905s) [noise=true , address=104.36.82.2:4731 , piece_id=PQWR56TKNZFB6KC5UBKIG5RIJMTHHUZOFPHJC5AUEO7N4SRRPTVA, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 159.223.143.39:49002->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.213148364 (1h26m45.99408173s) [noise=true , address=104.36.82.5:4838 , piece_id=ERVCAMGCRELCFK4N3Z7XDTI57O7YDIL2DYLJIZW6JE3CNVNY3IJQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:55074->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.213171019 (1h26m45.994104404s) [seg_pos={0 0}, upload=1330]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:18:30.247278023 (1h26m46.028211403s) [upload=1330]: Done uploading segments... Jul 26 23:18:30.247309019 (1h26m46.028242397s) [upload=1330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.049746186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXX] duration: 1m0.051942089s FAIL (2025-07-26 23:18:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== (1h25m33.679276158s) [seg_pos={0 0}, upload=1321]: Begin upload segment... Jul 26 23:17:17.898487866 (1h25m33.67942128s) [noise=true , address=77.90.31.18:4837 , piece_id=SR4GCYSEDIH3CRH7KNWQIKIXKOZK2NRTC4MZLKYXFHZO7ESO25VA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.899136989 (1h25m33.680070386s) [seg_pos={0 0}, upload=1321]: Done begin upload segment. Jul 26 23:17:17.899207466 (1h25m33.680140851s) [noise=true , address=64.49.13.170:4721 , piece_id=WWK27EQYJYGWBTOHF3B6ZXAOMJQ2HKNUOXYXGXPKTX62BEEOBYCQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.899447357 (1h25m33.680380753s) [noise=true , address=79.127.205.235:4688 , piece_id=FY4PM3YJ5AQROIMCOBJ2P2DU4AHZ2SPMQTVOQTFNASV6PC4NZQVA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.899650562 (1h25m33.680583929s) [noise=true , address=77.90.31.15:4703 , piece_id=4TOASMN3ABGPOZCAKYADVEVTXCXKCCJI76H7EPQ3ZFU7NJRU66AA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.900182475 (1h25m33.681115864s) [noise=true , address=45.91.116.5:4307 , piece_id=FBQOGXKFKZG6RYX45CEJJDUYBKY7USRFSM5GST42ODASUSMVXXAA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.900432091 (1h25m33.681365521s) [noise=true , address=104.36.82.56:4255 , piece_id=XM4JONWFX7UYNOIZFVMBP5DXAOJCUSFYVKAUECFZQTUJBDONGVZA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.900750299 (1h25m33.681683686s) [noise=true , address=104.36.82.47:4542 , piece_id=JADKC4MFZAOPUPL3BRXDDOLUKIEXCN2F5MQKVAB2MFTZT6GFFOFQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.900825191 (1h25m33.681758568s) [noise=true , address=104.36.82.46:4492 , piece_id=L7CWCXATU6F4NWE6DKN6XGQYUX5732IIRTBLLCQISHPQMTSTVIOA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.901129414 (1h25m33.682062791s) [noise=true , address=45.91.116.11:4151 , piece_id=FFL5B7C2WYTVCZNJHAZCBYJJB7NBXVHZLA7YRXRHNNFCGQUX4I2Q, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.901233930 (1h25m33.682167317s) [noise=true , address=77.90.31.11:4575 , piece_id=V7OFPGJJIJS364AV4DU5PHDEUIXQFSWSJOADV6BY4NPDF255HKLQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.901432735 (1h25m33.682366111s) [noise=true , address=64.49.13.140:4508 , piece_id=7CSAZQHUZEG33LD3B5NGKNGCJZ37UUBQMP7F276D5GEJDDUFMENA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.901722834 (1h25m33.682656211s) [noise=true , address=104.36.82.53:4138 , piece_id=USPNABGWEQ7BR2KSGPHBIECOTITHZK2SNULGSFYYSWNGPM2SGZUA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.903061321 (1h25m33.683994716s) [noise=true , address=104.36.82.14:4235 , piece_id=BCLYC7SJUICCD32S7GOAAGZH7O7JOC7AVX3QRKW5FZLXANXRX6BQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.903803835 (1h25m33.684737226s) [noise=true , address=156.146.43.225:4434 , piece_id=3WFGSWUFBLHIQE6QAYBPCL7EESTJEWSVKZBQXAVLF3FBDBTE7HAQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.904644102 (1h25m33.68557749s) [noise=true , address=64.49.13.138:4482 , piece_id=CTYW2QA4KK3Q6IOB3IVW26WE7PSAUQQTVEGEC6R6IVNXGQLLRY2Q, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.905922425 (1h25m33.686855822s) [noise=true , address=77.90.31.16:4764 , piece_id=KWHZ4B5HIWIULJ7Q6AZIAYJ3MA56EGCNZNNIBGYIXKD46EPHPYYQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.907256071 (1h25m33.68818946s) [noise=true , address=45.91.116.14:4206 , piece_id=5F23262FGOUQTTTYPTL3SN3ZNUWKBBCJ76EK526OHPEIAIV4OFSQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.907843336 (1h25m33.688776726s) [noise=true , address=104.36.82.5:4868 , piece_id=FKKNGX3TZ3K2UFRRTW7XO2MAXFSSSP7UNSIE5ZJQDF2DX5TFGAAQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.980250731 (1h25m33.761184104s) [upload=1321]: Next returned nil segment Jul 26 23:17:17.980272140 (1h25m33.761205518s) [upload=1321]: Waiting for error group managing segments... Jul 26 23:17:18.005533130 (1h25m33.78646651s) [noise=true , address=77.90.31.18:4837 , piece_id=SR4GCYSEDIH3CRH7KNWQIKIXKOZK2NRTC4MZLKYXFHZO7ESO25VA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:18.023420123 (1h25m33.8043535s) [noise=true , address=77.90.31.15:4703 , piece_id=4TOASMN3ABGPOZCAKYADVEVTXCXKCCJI76H7EPQ3ZFU7NJRU66AA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:18.076351467 (1h25m33.857284831s) [noise=true , address=77.90.31.11:4575 , piece_id=V7OFPGJJIJS364AV4DU5PHDEUIXQFSWSJOADV6BY4NPDF255HKLQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:18.088522435 (1h25m33.869455813s) [noise=true , address=77.90.31.16:4764 , piece_id=KWHZ4B5HIWIULJ7Q6AZIAYJ3MA56EGCNZNNIBGYIXKD46EPHPYYQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:18.482608960 (1h25m34.263542323s) [noise=true , address=79.127.205.235:4688 , piece_id=FY4PM3YJ5AQROIMCOBJ2P2DU4AHZ2SPMQTVOQTFNASV6PC4NZQVA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:19.494374274 (1h25m35.275307637s) [noise=true , address=45.91.116.5:4307 , piece_id=FBQOGXKFKZG6RYX45CEJJDUYBKY7USRFSM5GST42ODASUSMVXXAA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:19.749757663 (1h25m35.530691042s) [noise=true , address=45.91.116.11:4151 , piece_id=FFL5B7C2WYTVCZNJHAZCBYJJB7NBXVHZLA7YRXRHNNFCGQUX4I2Q, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: storage node overloaded, request limit: 2000 Jul 26 23:17:19.954329893 (1h25m35.735263274s) [noise=true , address=64.49.13.138:4482 , piece_id=CTYW2QA4KK3Q6IOB3IVW26WE7PSAUQQTVEGEC6R6IVNXGQLLRY2Q, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:17:21.016259136 (1h25m36.797192502s) [noise=true , address=156.146.43.225:4434 , piece_id=3WFGSWUFBLHIQE6QAYBPCL7EESTJEWSVKZBQXAVLF3FBDBTE7HAQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:21.697199166 (1h25m37.478132539s) [noise=true , address=64.49.13.170:4721 , piece_id=WWK27EQYJYGWBTOHF3B6ZXAOMJQ2HKNUOXYXGXPKTX62BEEOBYCQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:22.733238920 (1h25m38.51417229s) [noise=true , address=45.91.116.14:4206 , piece_id=5F23262FGOUQTTTYPTL3SN3ZNUWKBBCJ76EK526OHPEIAIV4OFSQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:24.554326344 (1h25m40.335259726s) [noise=true , address=64.49.13.140:4508 , piece_id=7CSAZQHUZEG33LD3B5NGKNGCJZ37UUBQMP7F276D5GEJDDUFMENA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:18:17.840751105 (1h26m33.621684499s) [noise=true , address=104.36.82.47:4542 , piece_id=JADKC4MFZAOPUPL3BRXDDOLUKIEXCN2F5MQKVAB2MFTZT6GFFOFQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: rpc: dial tcp 104.36.82.47:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.840904888 (1h26m33.621838269s) [noise=true , address=104.36.82.46:4492 , piece_id=L7CWCXATU6F4NWE6DKN6XGQYUX5732IIRTBLLCQISHPQMTSTVIOA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip, address:104.36.82.46:4492): protocol: rpc: dial tcp 104.36.82.46:4492: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.840979980 (1h26m33.621913359s) [noise=true , address=104.36.82.14:4235 , piece_id=BCLYC7SJUICCD32S7GOAAGZH7O7JOC7AVX3QRKW5FZLXANXRX6BQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV, address:104.36.82.14:4235): protocol: rpc: dial tcp 104.36.82.14:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.841056841 (1h26m33.621990227s) [noise=true , address=104.36.82.5:4868 , piece_id=FKKNGX3TZ3K2UFRRTW7XO2MAXFSSSP7UNSIE5ZJQDF2DX5TFGAAQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:18Srfics6dFkQeHZ2WF5YkFuLm2719Pw4Xgg9ntk12KnTBcn8v, address:104.36.82.5:4868): protocol: rpc: dial tcp 104.36.82.5:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.841115018 (1h26m33.622048399s) [noise=true , address=104.36.82.53:4138 , piece_id=USPNABGWEQ7BR2KSGPHBIECOTITHZK2SNULGSFYYSWNGPM2SGZUA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: rpc: dial tcp 104.36.82.53:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.841184398 (1h26m33.622117783s) [noise=true , address=104.36.82.56:4255 , piece_id=XM4JONWFX7UYNOIZFVMBP5DXAOJCUSFYVKAUECFZQTUJBDONGVZA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:12S6rwimXBvaEb4KneAwL5H3t5zTmvGVhyJSWCTU1u17r4RWJta, address:104.36.82.56:4255): protocol: write tcp 159.223.143.39:42970->104.36.82.56:4255: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.841206883 (1h26m33.622140269s) [seg_pos={0 0}, upload=1321]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:18:17.872061485 (1h26m33.652994901s) [upload=1321]: Done uploading segments... Jul 26 23:18:17.872090760 (1h26m33.65302414s) [upload=1321]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.052055567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..XX.................X.X..........XX...X...X..XX.X...X..X...] duration: 35.837615673s FAIL (2025-07-26 23:18:11 UTC) context deadline exceeded Duration: 35.837638397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 16 out of 60 is failed (threshold=3) [..............X........................X......XXXXXXXXXXXXXX] duration: 30.035812192s FAIL (2025-07-26 23:18:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.035840949s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...XX.................X.X..........XX...X...X..XX.X...X..X..] duration: 52.078966636s FAIL (2025-07-26 23:17:35 UTC) context deadline exceeded Duration: 52.079279103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................................XXX.X....XX.X.] duration: 11.591963168s FAIL (2025-07-26 23:17:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.592019615s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [...............X........................X......XXXXXXXXXXXXX] duration: 30.042255229s FAIL (2025-07-26 23:17:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.042301243s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 30.065235507s FAIL (2025-07-26 23:17:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 58384044s) [upload=1311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:16:50.477460690 (1h25m6.258394074s) [upload=1311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:16:50.517769268 (1h25m6.298702652s) [seg_pos={0 0}, upload=1311]: Begin upload segment... Jul 26 23:16:50.517837176 (1h25m6.29877054s) [seg_pos={0 0}, upload=1311]: Done begin upload segment. Jul 26 23:16:50.517892715 (1h25m6.298826099s) [noise=true , address=45.91.116.5:4297 , piece_id=LYURRVXF2R77VFR3ZUCOV7FHIYJ6FVJBVCXZWWJ7I4XDS2WGJ6TQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.517904688 (1h25m6.298838077s) [noise=true , address=77.90.31.15:4698 , piece_id=YRE3VN2EOEUKY2FDBOLD4YPGFRFJIQE6V4ZJN5XOWK4DFZMIQP7Q, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.518417914 (1h25m6.299351302s) [noise=true , address=104.36.82.5:4840 , piece_id=SWINKVYHBBNUC2PURNJAKEYMLWYSBOA54FTSOHHCYN6V5IJYGA7Q, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.518532238 (1h25m6.299465625s) [noise=true , address=45.91.116.7:4011 , piece_id=N2AUHHVGEUKSIZA2R245GN5VRKDZK3FVQHTCOSONLILLLP6SRHWA, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.518929159 (1h25m6.299862542s) [noise=true , address=104.36.82.16:4293 , piece_id=HLG6C6RLI66UU3QKM4VXMQNUAZCKKULDWA6RCRET4KB73SYEMQFA, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.519425318 (1h25m6.300358702s) [noise=true , address=104.36.82.10:4079 , piece_id=GDB44HDEYZ244SCHSQLK3QMUDHBTDXCEODET7LPLBHRABE4JAZ3A, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.519843567 (1h25m6.300776951s) [noise=true , address=104.36.82.2:4732 , piece_id=PYP46NTVETKN5RQYGKPJHXXCUAKAL5RURDHTNGY3NPVKSDFVPMIA, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.521088839 (1h25m6.302022227s) [noise=true , address=104.36.82.6:4873 , piece_id=TTAITX3FCLOT7CJAVREZZQTMMVW2PTS43KZAHZAQ7TQJTYWS2MXQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.521274355 (1h25m6.302207743s) [noise=true , address=104.36.82.10:4078 , piece_id=LPDC6PQHHZ6AD7OM6H3Z6MRV5CDCF7RK6AOHJBX7AUG5VKBWLABQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.521728851 (1h25m6.302662241s) [noise=true , address=45.91.116.6:4975 , piece_id=4DOIAVU55WK46W6QSZPX2RHHSVAUBGQA5OVKWJJXCHBJYBXZ6L2Q, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.522203238 (1h25m6.303136641s) [noise=true , address=45.91.116.4:4940 , piece_id=WSMBJJGA6BGNV3KTN54W7T5MUFFID5XEAEWI3OSUSUPOA7UIAZYQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.522645516 (1h25m6.303578902s) [noise=true , address=104.36.82.15:4257 , piece_id=LH5O7SDUZ7LYHRAU55GZ73HX4KYLWHNH2KVVM6EVJZKBXMAPISFQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.523105751 (1h25m6.304039135s) [noise=true , address=104.36.82.9:4006 , piece_id=C3FWR7LAKHX77EY2MRCJHXHKRUDU2POJWD34MRYXSKC5JSV4L54Q, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.523528756 (1h25m6.304462143s) [noise=true , address=104.36.82.12:4151 , piece_id=DQEKMH7QGF5X2775A7UYL3IEJISGCB4OFKC72VJMCOZUXDH5I6QA, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.523959751 (1h25m6.304893138s) [noise=true , address=45.91.116.12:4227 , piece_id=TGYFKF6FDIDTHKXKFO35LYFRCIAPSRUG2WQKT6LKK2R6WAJGR2GQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.524394046 (1h25m6.305327418s) [noise=true , address=104.36.82.15:4259 , piece_id=V3OG6ROGD3ULBBJQLLKZXGGQU236P2RRX2MHGU3NYHA5YZUOWAAA, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.624846792 (1h25m6.405780182s) [upload=1311]: Next returned nil segment Jul 26 23:16:50.624873984 (1h25m6.405807376s) [upload=1311]: Waiting for error group managing segments... Jul 26 23:16:50.629573272 (1h25m6.410506638s) [noise=true , address=45.91.116.7:4011 , piece_id=N2AUHHVGEUKSIZA2R245GN5VRKDZK3FVQHTCOSONLILLLP6SRHWA, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:16:50.630179183 (1h25m6.41111256s) [noise=true , address=45.91.116.6:4975 , piece_id=4DOIAVU55WK46W6QSZPX2RHHSVAUBGQA5OVKWJJXCHBJYBXZ6L2Q, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:16:50.630712922 (1h25m6.4116463s) [noise=true , address=45.91.116.4:4940 , piece_id=WSMBJJGA6BGNV3KTN54W7T5MUFFID5XEAEWI3OSUSUPOA7UIAZYQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:16:50.631271461 (1h25m6.412204842s) [noise=true , address=45.91.116.12:4227 , piece_id=TGYFKF6FDIDTHKXKFO35LYFRCIAPSRUG2WQKT6LKK2R6WAJGR2GQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:16:50.631865820 (1h25m6.412799191s) [noise=true , address=45.91.116.5:4297 , piece_id=LYURRVXF2R77VFR3ZUCOV7FHIYJ6FVJBVCXZWWJ7I4XDS2WGJ6TQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:16:50.648254688 (1h25m6.429188072s) [noise=true , address=77.90.31.15:4698 , piece_id=YRE3VN2EOEUKY2FDBOLD4YPGFRFJIQE6V4ZJN5XOWK4DFZMIQP7Q, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:17:20.439481673 (1h25m36.220415038s) [noise=true , address=104.36.82.2:4732 , piece_id=PYP46NTVETKN5RQYGKPJHXXCUAKAL5RURDHTNGY3NPVKSDFVPMIA, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.439631072 (1h25m36.220564434s) [noise=true , address=104.36.82.9:4006 , piece_id=C3FWR7LAKHX77EY2MRCJHXHKRUDU2POJWD34MRYXSKC5JSV4L54Q, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.439730850 (1h25m36.220664227s) [noise=true , address=104.36.82.10:4079 , piece_id=GDB44HDEYZ244SCHSQLK3QMUDHBTDXCEODET7LPLBHRABE4JAZ3A, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.439805025 (1h25m36.220738403s) [noise=true , address=104.36.82.15:4259 , piece_id=V3OG6ROGD3ULBBJQLLKZXGGQU236P2RRX2MHGU3NYHA5YZUOWAAA, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.439867887 (1h25m36.220801263s) [noise=true , address=104.36.82.6:4873 , piece_id=TTAITX3FCLOT7CJAVREZZQTMMVW2PTS43KZAHZAQ7TQJTYWS2MXQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:58148->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.440290437 (1h25m36.221223822s) [noise=true , address=104.36.82.10:4078 , piece_id=LPDC6PQHHZ6AD7OM6H3Z6MRV5CDCF7RK6AOHJBX7AUG5VKBWLABQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.440459329 (1h25m36.221392706s) [noise=true , address=104.36.82.15:4257 , piece_id=LH5O7SDUZ7LYHRAU55GZ73HX4KYLWHNH2KVVM6EVJZKBXMAPISFQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:57526->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.440588750 (1h25m36.221522127s) [noise=true , address=104.36.82.12:4151 , piece_id=DQEKMH7QGF5X2775A7UYL3IEJISGCB4OFKC72VJMCOZUXDH5I6QA, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.440697571 (1h25m36.221630948s) [noise=true , address=104.36.82.5:4840 , piece_id=SWINKVYHBBNUC2PURNJAKEYMLWYSBOA54FTSOHHCYN6V5IJYGA7Q, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.440815859 (1h25m36.221749236s) [noise=true , address=104.36.82.16:4293 , piece_id=HLG6C6RLI66UU3QKM4VXMQNUAZCKKULDWA6RCRET4KB73SYEMQFA, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:48332->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.441167675 (1h25m36.222101065s) [seg_pos={0 0}, upload=1311]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:17:20.501459163 (1h25m36.282392549s) [upload=1311]: Done uploading segments... Jul 26 23:17:20.501484131 (1h25m36.282417522s) [upload=1311]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.065254163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXX] duration: 1m0.038425369s FAIL (2025-07-26 23:17:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== equestItem_SegmentBegin Jul 26 23:16:15.268298842 (1h24m31.049232227s) [seg_pos={0 0}, upload=1302]: Begin upload segment... Jul 26 23:16:15.268365966 (1h24m31.049299347s) [seg_pos={0 0}, upload=1302]: Done begin upload segment. Jul 26 23:16:15.268404721 (1h24m31.049338096s) [noise=true , address=104.36.82.56:4260 , piece_id=2PGOD55EA2ED3W56OQKYJ6UIV2G6XZTNDGTHZDM56XRXUUEBBBEA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.268412734 (1h24m31.049346115s) [noise=true , address=104.36.82.47:4545 , piece_id=L4XB777Z2VY5XJVMTJVXEAO5EK4T6CCJAHJ3Y6CQQRSA7X2H5YCA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.268893545 (1h24m31.049826923s) [noise=true , address=64.49.13.170:4721 , piece_id=JIAUBN26UC7BTKX7T4GUISAY74IENIABVFQKRRZNJNA7SHHWOOBA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.268926288 (1h24m31.049859668s) [noise=true , address=77.90.31.15:4726 , piece_id=MS4CIZJDWVCN27OEBEG6N4CEQ5TOW6456WDBMG6TUSHWCFUX3DVA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.269365569 (1h24m31.050298955s) [noise=true , address=104.36.82.52:4101 , piece_id=Y6HDBTTEI4HNFN7TB4ZEYOEVCARJ55MRFZHW4ARGVD2NO3XFJACA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.269419395 (1h24m31.05035277s) [noise=true , address=79.127.205.235:4682 , piece_id=5SS3RY4IEULQJFTRKBMLBEARQ67IFFEQVWKUL66T4CH3C44HMHFA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.269813259 (1h24m31.050746643s) [noise=true , address=79.127.205.245:4226 , piece_id=SG3ZOSR3LRKINCJFCMV56WLLKZL5HZH46UEQ2MFMMPXK6ZODCGFA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.269893057 (1h24m31.050826438s) [noise=true , address=104.36.82.11:4123 , piece_id=RCY7OFL3U5HF4UC22ODNKOLUTPAW2HL6IONKBYHMKD2F3TVXDHPA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.270239108 (1h24m31.051172492s) [noise=true , address=156.146.43.225:4449 , piece_id=ISQYFQRC57ANVPEBP5JXB6UMX7EAGDTIU25CFH523HRA4WMMES6Q, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.270392634 (1h24m31.05132602s) [noise=true , address=77.90.31.18:4821 , piece_id=C5VUWYWF2UNRPA6Z2RW7HTCSU3JL23RR7MVCHAGOW6AZHENAFP6A, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.270653190 (1h24m31.051586578s) [noise=true , address=79.127.201.215:4345 , piece_id=OA25U765MWY5IWCRLT4MYZ7MWQ4XGFRQXR6KHGLIKMTQ27PKIW5Q, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.270965440 (1h24m31.051898827s) [noise=true , address=79.127.205.234:4609 , piece_id=FUSNJM2K3NH6C557SNAJEWGGUAJSLYC7HO2CZA4UJPGI4AQEERPA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.271109225 (1h24m31.052042609s) [noise=true , address=104.36.82.55:4201 , piece_id=H7TDKFHQFUIIPR5KCXA664M5NEIKJJV432X2CBZQMS5YEJV7MWGQ, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.271547916 (1h24m31.052481339s) [noise=true , address=104.36.82.46:4495 , piece_id=2AFBZA3Z3L2LLMVHUCCCZZYQPMB7NNVAU7S7ZRQJ7O4WFNGCKNBQ, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.271579270 (1h24m31.052512647s) [noise=true , address=104.36.82.54:4173 , piece_id=AENJ36PKON2ZH23NUW2UZO3NYGWYQO4WSR7ZPZPZOKQY25XOHKEA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.272264283 (1h24m31.053197665s) [noise=true , address=77.90.31.11:4569 , piece_id=B77OVX2LDCZRWB23G7X26KYA25QB6M665O7GNB2SXSLJD2CUYHBQ, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.272699471 (1h24m31.053632856s) [noise=true , address=64.49.13.140:4508 , piece_id=LMSJM6GA35M2Y4YLGIAHEMQWU2MRXBYUKFNEOMFAKUXO7ACXQPOA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.273635509 (1h24m31.054568878s) [noise=true , address=79.127.205.233:4647 , piece_id=KBD2YWTEXL64UFR6SM7J6JR7GTTO26UNJJRIYAMLJTBOJTSVAEHQ, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.347248383 (1h24m31.128181765s) [upload=1302]: Next returned nil segment Jul 26 23:16:15.347271400 (1h24m31.128204778s) [upload=1302]: Waiting for error group managing segments... Jul 26 23:16:15.385700175 (1h24m31.166633555s) [noise=true , address=77.90.31.15:4726 , piece_id=MS4CIZJDWVCN27OEBEG6N4CEQ5TOW6456WDBMG6TUSHWCFUX3DVA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.440787186 (1h24m31.221720547s) [noise=true , address=77.90.31.11:4569 , piece_id=B77OVX2LDCZRWB23G7X26KYA25QB6M665O7GNB2SXSLJD2CUYHBQ, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.532146658 (1h24m31.313080036s) [noise=true , address=77.90.31.18:4821 , piece_id=C5VUWYWF2UNRPA6Z2RW7HTCSU3JL23RR7MVCHAGOW6AZHENAFP6A, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.862871119 (1h24m31.643804491s) [noise=true , address=79.127.205.245:4226 , piece_id=SG3ZOSR3LRKINCJFCMV56WLLKZL5HZH46UEQ2MFMMPXK6ZODCGFA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.868901356 (1h24m31.649834736s) [noise=true , address=79.127.205.234:4609 , piece_id=FUSNJM2K3NH6C557SNAJEWGGUAJSLYC7HO2CZA4UJPGI4AQEERPA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.879573398 (1h24m31.660506767s) [noise=true , address=79.127.205.233:4647 , piece_id=KBD2YWTEXL64UFR6SM7J6JR7GTTO26UNJJRIYAMLJTBOJTSVAEHQ, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.904954137 (1h24m31.685887506s) [noise=true , address=79.127.205.235:4682 , piece_id=5SS3RY4IEULQJFTRKBMLBEARQ67IFFEQVWKUL66T4CH3C44HMHFA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:21.848372982 (1h24m37.629306343s) [noise=true , address=64.49.13.140:4508 , piece_id=LMSJM6GA35M2Y4YLGIAHEMQWU2MRXBYUKFNEOMFAKUXO7ACXQPOA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:22.380953153 (1h24m38.161886517s) [noise=true , address=156.146.43.225:4449 , piece_id=ISQYFQRC57ANVPEBP5JXB6UMX7EAGDTIU25CFH523HRA4WMMES6Q, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:23.076539544 (1h24m38.85747292s) [noise=true , address=64.49.13.170:4721 , piece_id=JIAUBN26UC7BTKX7T4GUISAY74IENIABVFQKRRZNJNA7SHHWOOBA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:53.470850199 (1h25m9.251783563s) [noise=true , address=79.127.201.215:4345 , piece_id=OA25U765MWY5IWCRLT4MYZ7MWQ4XGFRQXR6KHGLIKMTQ27PKIW5Q, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:17:15.201468483 (1h25m30.982401861s) [noise=true , address=104.36.82.55:4201 , piece_id=H7TDKFHQFUIIPR5KCXA664M5NEIKJJV432X2CBZQMS5YEJV7MWGQ, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.201610806 (1h25m30.982544185s) [noise=true , address=104.36.82.46:4495 , piece_id=2AFBZA3Z3L2LLMVHUCCCZZYQPMB7NNVAU7S7ZRQJ7O4WFNGCKNBQ, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:1YpHarCJ3ouMGTs59udhU7az9h68ELEHTa2aTAax2chR2Gux1x, address:104.36.82.46:4495): protocol: rpc: dial tcp 104.36.82.46:4495: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.201720134 (1h25m30.982653508s) [noise=true , address=104.36.82.56:4260 , piece_id=2PGOD55EA2ED3W56OQKYJ6UIV2G6XZTNDGTHZDM56XRXUUEBBBEA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: rpc: dial tcp 104.36.82.56:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.201750754 (1h25m30.982684138s) [noise=true , address=104.36.82.47:4545 , piece_id=L4XB777Z2VY5XJVMTJVXEAO5EK4T6CCJAHJ3Y6CQQRSA7X2H5YCA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: write tcp 159.223.143.39:49546->104.36.82.47:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.201793937 (1h25m30.982727315s) [noise=true , address=104.36.82.11:4123 , piece_id=RCY7OFL3U5HF4UC22ODNKOLUTPAW2HL6IONKBYHMKD2F3TVXDHPA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:1LajLoxqMcmnKSCX6hVkthYVKZKTsPGphQHerwyjv4pi5GrXSE, address:104.36.82.11:4123): protocol: rpc: dial tcp 104.36.82.11:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.201949074 (1h25m30.982882458s) [noise=true , address=104.36.82.54:4173 , piece_id=AENJ36PKON2ZH23NUW2UZO3NYGWYQO4WSR7ZPZPZOKQY25XOHKEA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: rpc: dial tcp 104.36.82.54:4173: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.202335833 (1h25m30.983269213s) [noise=true , address=104.36.82.52:4101 , piece_id=Y6HDBTTEI4HNFN7TB4ZEYOEVCARJ55MRFZHW4ARGVD2NO3XFJACA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: write tcp 159.223.143.39:51004->104.36.82.52:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.202365325 (1h25m30.983298709s) [seg_pos={0 0}, upload=1302]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:17:15.234625329 (1h25m31.015558738s) [upload=1302]: Done uploading segments... Jul 26 23:17:15.234657510 (1h25m31.015590891s) [upload=1302]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.038468151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 30.026246927s FAIL (2025-07-26 23:16:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ding segments... Jul 26 23:16:20.280878777 (1h24m36.061812156s) [upload=1298]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:16:20.375333897 (1h24m36.156267258s) [upload=1305]: Starting upload Jul 26 23:16:20.375397763 (1h24m36.156331152s) [upload=1305]: Done starting upload Jul 26 23:16:20.375721527 (1h24m36.156654896s) [upload=1305]: Uploading segments... Jul 26 23:16:20.375787002 (1h24m36.156720381s) [upload=1305]: Got next segment. Inline: false Jul 26 23:16:20.375834274 (1h24m36.15676764s) [upload=1305]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:16:20.375839624 (1h24m36.15677299s) [upload=1305]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:16:20.403893360 (1h24m36.184826741s) [seg_pos={0 0}, upload=1305]: Begin upload segment... Jul 26 23:16:20.404128657 (1h24m36.18506202s) [seg_pos={0 0}, upload=1305]: Done begin upload segment. Jul 26 23:16:20.404164608 (1h24m36.185097989s) [noise=true , address=77.90.31.16:4735 , piece_id=EWKB67RZRW73IKGZPFGFHEVIXEKHXGPG56GBIYUD4LN75LLY2HGA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.404741840 (1h24m36.185675227s) [noise=true , address=77.90.31.14:4663 , piece_id=K4BHKWIO6CBXDLGCU5B6DOEEDBFDETNL24VNVE4HT6ME237HKW4Q, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.405206720 (1h24m36.186140108s) [noise=true , address=104.36.82.2:4730 , piece_id=JA32EI23QHJY3DS4TWRGHE56KE4NIKISA6ESYXFQ43TRH65EOJVQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.405691454 (1h24m36.186624841s) [noise=true , address=104.36.82.14:4223 , piece_id=ABARSEDKB6U445H6P65JDKII6YLQWKSWU6WM5HQZMF6EXRYTGLEA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.406149221 (1h24m36.187082609s) [noise=true , address=45.91.116.14:4190 , piece_id=IGG2UAWUPLGP7WPBUEP7WEPS7NK375QR34ZVGVQR5XHHDGHWXQPQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.406623605 (1h24m36.187557065s) [noise=true , address=45.91.116.4:4940 , piece_id=UB6UOLKRNMGH3X2OVYWNRIONTCZSGODCQ7ZDL6KEDAVW77JDJI6A, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.407406299 (1h24m36.188339687s) [noise=true , address=104.36.82.3:4765 , piece_id=FI4TTRKG4YNUWMZNFKQREBGUXLYYOZCLYAOUQEUVFMNIISSPIEZA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.408202208 (1h24m36.189135661s) [noise=true , address=104.36.82.7:4909 , piece_id=FNM3EIUZ3BF5RXOOU2PDTVDZGKOVBQJ3EPGHYIPBBV7HXUFIOMJQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.408995737 (1h24m36.189929122s) [noise=true , address=104.36.82.15:4259 , piece_id=3MB2GZ5THYJCLTMKEQ77BPQ4O37HJDKJNVVGTXEK2XE44HS6PPSA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.409487995 (1h24m36.190421428s) [noise=true , address=104.36.82.6:4873 , piece_id=MGRLY4IEVF7G2A2MKVPOWEBSGIF6GU6JIUBUGNFLZAWEPLLJHW2A, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.410122507 (1h24m36.191055895s) [noise=true , address=77.90.31.11:4553 , piece_id=MEZ74DFNRO5OJP6EIKK4H2YJHCNXAWBKVXQ6ZACOONVJK6PY43RA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.410662539 (1h24m36.191595925s) [noise=true , address=45.91.116.6:4975 , piece_id=KJTYAOJMGYKCC2WSWQVX2DODQ4KB3TAYPK6DTXXQQSPVWRSZPOGQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.411171041 (1h24m36.192104427s) [noise=true , address=45.91.116.6:4974 , piece_id=GOHPI3CUCV3YL432BTC6FL2TEQ2YKO4GU4IW6JVYN4MNTWKFPTRA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.411660403 (1h24m36.192593791s) [noise=true , address=45.91.116.11:4119 , piece_id=56NBVBYRPOTBTGOD4BEMZNSIUWKUZM566THTZBZGZP5JCC2I33FQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.412119588 (1h24m36.193052971s) [noise=true , address=104.36.82.4:4803 , piece_id=VHI36DWRX4UISAKQEGDEKYQNMPYDV5AIPCP2EDC2GYQMN4XTHVNA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.412549647 (1h24m36.193483036s) [noise=true , address=45.91.116.5:4299 , piece_id=R7NFCL6DRY3RNGTBWBD2I3V5W7GO4U7CLWMA2MJ3ZTK5CSOHUEIQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.499920520 (1h24m36.280853898s) [upload=1305]: Next returned nil segment Jul 26 23:16:20.499940464 (1h24m36.280873851s) [upload=1305]: Waiting for error group managing segments... Jul 26 23:16:20.505050235 (1h24m36.285983603s) [noise=true , address=45.91.116.5:4299 , piece_id=R7NFCL6DRY3RNGTBWBD2I3V5W7GO4U7CLWMA2MJ3ZTK5CSOHUEIQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.505127778 (1h24m36.286064938s) [noise=true , address=45.91.116.4:4940 , piece_id=UB6UOLKRNMGH3X2OVYWNRIONTCZSGODCQ7ZDL6KEDAVW77JDJI6A, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.505792186 (1h24m36.286725571s) [noise=true , address=77.90.31.11:4553 , piece_id=MEZ74DFNRO5OJP6EIKK4H2YJHCNXAWBKVXQ6ZACOONVJK6PY43RA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.505808539 (1h24m36.286741919s) [noise=true , address=45.91.116.14:4190 , piece_id=IGG2UAWUPLGP7WPBUEP7WEPS7NK375QR34ZVGVQR5XHHDGHWXQPQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.506377159 (1h24m36.28731054s) [noise=true , address=45.91.116.6:4975 , piece_id=KJTYAOJMGYKCC2WSWQVX2DODQ4KB3TAYPK6DTXXQQSPVWRSZPOGQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.506410063 (1h24m36.287343442s) [noise=true , address=45.91.116.6:4974 , piece_id=GOHPI3CUCV3YL432BTC6FL2TEQ2YKO4GU4IW6JVYN4MNTWKFPTRA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.509090453 (1h24m36.290023823s) [noise=true , address=77.90.31.14:4663 , piece_id=K4BHKWIO6CBXDLGCU5B6DOEEDBFDETNL24VNVE4HT6ME237HKW4Q, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.543011771 (1h24m36.323945133s) [noise=true , address=45.91.116.11:4119 , piece_id=56NBVBYRPOTBTGOD4BEMZNSIUWKUZM566THTZBZGZP5JCC2I33FQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:22.540735151 (1h24m38.321668513s) [noise=true , address=77.90.31.16:4735 , piece_id=EWKB67RZRW73IKGZPFGFHEVIXEKHXGPG56GBIYUD4LN75LLY2HGA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:50.344938979 (1h25m6.125872362s) [noise=true , address=104.36.82.15:4259 , piece_id=3MB2GZ5THYJCLTMKEQ77BPQ4O37HJDKJNVVGTXEK2XE44HS6PPSA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:39740->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345028052 (1h25m6.125961431s) [noise=true , address=104.36.82.7:4909 , piece_id=FNM3EIUZ3BF5RXOOU2PDTVDZGKOVBQJ3EPGHYIPBBV7HXUFIOMJQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345277383 (1h25m6.126210766s) [noise=true , address=104.36.82.2:4730 , piece_id=JA32EI23QHJY3DS4TWRGHE56KE4NIKISA6ESYXFQ43TRH65EOJVQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:47894->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345399979 (1h25m6.126333455s) [noise=true , address=104.36.82.6:4873 , piece_id=MGRLY4IEVF7G2A2MKVPOWEBSGIF6GU6JIUBUGNFLZAWEPLLJHW2A, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:33380->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345408872 (1h25m6.126342244s) [noise=true , address=104.36.82.14:4223 , piece_id=ABARSEDKB6U445H6P65JDKII6YLQWKSWU6WM5HQZMF6EXRYTGLEA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:36980->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345486772 (1h25m6.126420151s) [noise=true , address=104.36.82.4:4803 , piece_id=VHI36DWRX4UISAKQEGDEKYQNMPYDV5AIPCP2EDC2GYQMN4XTHVNA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 159.223.143.39:59728->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345500922 (1h25m6.126434285s) [noise=true , address=104.36.82.3:4765 , piece_id=FI4TTRKG4YNUWMZNFKQREBGUXLYYOZCLYAOUQEUVFMNIISSPIEZA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:57008->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345572721 (1h25m6.126506105s) [seg_pos={0 0}, upload=1305]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:16:50.367347168 (1h25m6.14828055s) [upload=1305]: Done uploading segments... Jul 26 23:16:50.367399899 (1h25m6.148333283s) [upload=1305]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.026317707s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X...XX.................X.X..........XX...X...X..XX.X...X..X.] duration: 47.046638853s FAIL (2025-07-26 23:16:43 UTC) context deadline exceeded Duration: 47.046657492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................................XXX.X....XX.X] duration: 30.002753447s FAIL (2025-07-26 23:16:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002840455s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [................X........................X......XXXXXXXXXXXX] duration: 30.032847757s FAIL (2025-07-26 23:16:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.033667953s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 30.035113112s FAIL (2025-07-26 23:16:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [upload=1298]: Got next segment. Inline: false Jul 26 23:15:50.279294479 (1h24m6.060227845s) [upload=1298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:15:50.279301109 (1h24m6.060234475s) [upload=1298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:15:50.307215353 (1h24m6.088148739s) [seg_pos={0 0}, upload=1298]: Begin upload segment... Jul 26 23:15:50.307301268 (1h24m6.088234654s) [seg_pos={0 0}, upload=1298]: Done begin upload segment. Jul 26 23:15:50.307314555 (1h24m6.08824794s) [noise=true , address=104.36.82.12:4150 , piece_id=4C6WEQD6BBT6ISOKRBIRU52N4NN53EXB7KZCUXDI4JSRMYFISD6Q, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.307331328 (1h24m6.08826471s) [noise=true , address=45.91.116.12:4226 , piece_id=3LTAMSRKVUIKI2QWYRXL7M2WN4LM6Y2ENN2HUW664VGMSL7APHQA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.307749007 (1h24m6.088682386s) [noise=true , address=77.90.31.14:4661 , piece_id=LBO6OIPOMGJJTDSBOCPSCYDUDJGQZ62LSRBQJWT5AZYRMPDJQVDA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.307863730 (1h24m6.088797115s) [noise=true , address=45.91.116.7:4011 , piece_id=7WTYEPTXOSTTOXAWU2NDYDHZNNVXOFCMTJ2PRVGODONOZKBBWIPQ, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.308461848 (1h24m6.089395223s) [noise=true , address=45.91.116.7:4010 , piece_id=SOI32TFCVJGLSJXDWY6LGWC5FK27BEI4RGHGWWJNKJQMN4TCR7SA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.308831042 (1h24m6.089764422s) [noise=true , address=104.36.82.10:4078 , piece_id=IFRROODHXTEYW5O4WL3EI4IQDBNV7565NNXTJKEJDURMDXZ3FH3A, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.308999953 (1h24m6.089933343s) [noise=true , address=104.36.82.10:4077 , piece_id=YSNWTGCPOXRUBZUYFSM3MZV32PMFBTGYF6RG6XNQ7TAMLJFDZNJA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.309480441 (1h24m6.090413825s) [noise=true , address=45.91.116.8:4048 , piece_id=AIH34MBLZPOAK456VHFENXVDA573CFOCKULIYMCYJXQALK3XGTBA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.309891269 (1h24m6.090824655s) [noise=true , address=104.36.82.11:4115 , piece_id=XF5LDPF742ZZHBR7UV5K3AHA6VZZOZSZ6C2S37TSKVXSMRDH4BEQ, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.310260739 (1h24m6.09119412s) [noise=true , address=45.91.116.6:4974 , piece_id=2IMF6TBQQQ77BXRQBC6TW3UMCJO7JXGL225UPVH4NMCILVVDHHPQ, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.310406206 (1h24m6.091339591s) [noise=true , address=104.36.82.14:4223 , piece_id=X5EE6YWLNWGFPXXHXPT7MNRMSOXXN5LHKN42TJDUQIAA3QWEG3FA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.310996588 (1h24m6.091929972s) [noise=true , address=45.91.116.13:4153 , piece_id=OHJT6EFXAZFYKDLUDGX4D2UCNCZY2OY5RFOJY3SXVBYATUHB5YLQ, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.311275061 (1h24m6.092208445s) [noise=true , address=104.36.82.7:4909 , piece_id=VW4677D73Q55RF43MXJD4AHHWVA5F5VJG5F3XPHOVZBMCTC452DQ, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.311702084 (1h24m6.092635473s) [noise=true , address=104.36.82.6:4874 , piece_id=QG4IR55URTAASLOZP37BUFO7UQ7FZY6IMPVE3VW7MGM34HR4GJQA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.312105707 (1h24m6.093039115s) [noise=true , address=104.36.82.15:4259 , piece_id=HW6MC6IDQPAARIBLOAE6PXA3LCKXJ35ZSG4ZL63DYS7NHQ7KYFXA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.312610980 (1h24m6.093544362s) [noise=true , address=104.36.82.2:4730 , piece_id=5R3K6OBC23JFHU6TKZETT75OV7TUXDJQ4QYRH33K6LHZI7WZ4LEA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.377543304 (1h24m6.158476672s) [upload=1298]: Next returned nil segment Jul 26 23:15:50.377558334 (1h24m6.158491709s) [upload=1298]: Waiting for error group managing segments... Jul 26 23:15:50.381622959 (1h24m6.162556328s) [noise=true , address=45.91.116.7:4011 , piece_id=7WTYEPTXOSTTOXAWU2NDYDHZNNVXOFCMTJ2PRVGODONOZKBBWIPQ, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.381725768 (1h24m6.162659143s) [noise=true , address=45.91.116.8:4048 , piece_id=AIH34MBLZPOAK456VHFENXVDA573CFOCKULIYMCYJXQALK3XGTBA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.382198192 (1h24m6.163131602s) [noise=true , address=45.91.116.6:4974 , piece_id=2IMF6TBQQQ77BXRQBC6TW3UMCJO7JXGL225UPVH4NMCILVVDHHPQ, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.382234818 (1h24m6.163168187s) [noise=true , address=45.91.116.7:4010 , piece_id=SOI32TFCVJGLSJXDWY6LGWC5FK27BEI4RGHGWWJNKJQMN4TCR7SA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.382606982 (1h24m6.163540347s) [noise=true , address=45.91.116.13:4153 , piece_id=OHJT6EFXAZFYKDLUDGX4D2UCNCZY2OY5RFOJY3SXVBYATUHB5YLQ, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.382656693 (1h24m6.163590057s) [noise=true , address=45.91.116.12:4226 , piece_id=3LTAMSRKVUIKI2QWYRXL7M2WN4LM6Y2ENN2HUW664VGMSL7APHQA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.383093682 (1h24m6.164027048s) [noise=true , address=77.90.31.14:4661 , piece_id=LBO6OIPOMGJJTDSBOCPSCYDUDJGQZ62LSRBQJWT5AZYRMPDJQVDA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:16:20.247621196 (1h24m36.028554572s) [noise=true , address=104.36.82.12:4150 , piece_id=4C6WEQD6BBT6ISOKRBIRU52N4NN53EXB7KZCUXDI4JSRMYFISD6Q, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:39762->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.247708578 (1h24m36.028641955s) [noise=true , address=104.36.82.6:4874 , piece_id=QG4IR55URTAASLOZP37BUFO7UQ7FZY6IMPVE3VW7MGM34HR4GJQA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:35944->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.247829726 (1h24m36.028763092s) [noise=true , address=104.36.82.2:4730 , piece_id=5R3K6OBC23JFHU6TKZETT75OV7TUXDJQ4QYRH33K6LHZI7WZ4LEA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:55314->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.247898812 (1h24m36.028832179s) [noise=true , address=104.36.82.11:4115 , piece_id=XF5LDPF742ZZHBR7UV5K3AHA6VZZOZSZ6C2S37TSKVXSMRDH4BEQ, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:38640->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.248293937 (1h24m36.029227312s) [noise=true , address=104.36.82.14:4223 , piece_id=X5EE6YWLNWGFPXXHXPT7MNRMSOXXN5LHKN42TJDUQIAA3QWEG3FA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:57882->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.248374212 (1h24m36.02930758s) [noise=true , address=104.36.82.15:4259 , piece_id=HW6MC6IDQPAARIBLOAE6PXA3LCKXJ35ZSG4ZL63DYS7NHQ7KYFXA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:47310->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.248391018 (1h24m36.029324382s) [noise=true , address=104.36.82.10:4078 , piece_id=IFRROODHXTEYW5O4WL3EI4IQDBNV7565NNXTJKEJDURMDXZ3FH3A, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:58836->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.248425810 (1h24m36.029359176s) [noise=true , address=104.36.82.10:4077 , piece_id=YSNWTGCPOXRUBZUYFSM3MZV32PMFBTGYF6RG6XNQ7TAMLJFDZNJA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:35130->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.248538809 (1h24m36.029472185s) [noise=true , address=104.36.82.7:4909 , piece_id=VW4677D73Q55RF43MXJD4AHHWVA5F5VJG5F3XPHOVZBMCTC452DQ, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:16:20.248610185 (1h24m36.02954356s) [seg_pos={0 0}, upload=1298]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:16:20.280846396 (1h24m36.06177976s) [upload=1298]: Done uploading segments... Jul 26 23:16:20.280878777 (1h24m36.061812156s) [upload=1298]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.035145579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXX] duration: 1m0.030273335s FAIL (2025-07-26 23:16:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== pload=1287]: Uploading piece... Jul 26 23:15:07.972509291 (1h23m23.75344268s) [noise=true , address=104.36.82.11:4121 , piece_id=HUHD7H3OQD2IK43YMUVDEFXFXZAFP77FRFRBEF6IGX2OBZFEFYHA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.973080267 (1h23m23.754013697s) [noise=true , address=104.36.82.53:4122 , piece_id=FEUI7GYJZLYQWLTPF2MGUQGHU64XUIDVR3CS6K5LOIBQ75A3URYA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.973562022 (1h23m23.754495409s) [noise=true , address=45.91.116.9:4273 , piece_id=JVNX4GTSC54GRKPZVFLLQHC2KDZRC47KBP6BXQDMAL37K5XEIGGA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.974036552 (1h23m23.754969951s) [noise=true , address=79.127.205.235:4686 , piece_id=FQFJS5F3BZKRYMMY7DE7OJSIVMDG4HPQJLNOXYKWSZAMMSWJMRFA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.974562455 (1h23m23.755495863s) [noise=true , address=104.36.82.46:4506 , piece_id=67NG5XPCAIIYY53GEKDCCYRRJTQ3LDPELSFY2PXJT54A2IJCOEOA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975053871 (1h23m23.755987255s) [noise=true , address=79.127.205.246:4284 , piece_id=G3O7BHQJSM5OL6VKT5QOAF4VENG7IMOUPWQ5THV7MZHRONBDMJCA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975145452 (1h23m23.756078836s) [noise=true , address=45.91.116.11:4121 , piece_id=72UBINQFQAAAB5DJNRNHSNXDUZTN75NLIGWO2XQLLEXQUZFREZEA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975520326 (1h23m23.756453715s) [noise=true , address=104.36.82.52:4090 , piece_id=PNBVLIRI47XNQAAKEFSHUJ72WUXGHNQWVEOYH3PRXVDOULSZOH3Q, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975537033 (1h23m23.75647041s) [noise=true , address=104.36.82.55:4206 , piece_id=Q6I5CAH7S6CISDOY3EWWLZTXDISMLKJ3KUQFW2OXAS5GGF2D43BQ, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975856702 (1h23m23.756790078s) [noise=true , address=77.90.31.15:4713 , piece_id=B6BKAO2PCS2APNPYT2PNRFOFBDJ362YMGFTJNZ66JSNTIAPF6ZNA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975935292 (1h23m23.756868679s) [noise=true , address=64.49.13.140:4508 , piece_id=EXU4MEC7557C6G6F3LWBJC4PYVNH263E3AMCXMV6POQK22DRMTYQ, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.976330564 (1h23m23.757263944s) [noise=true , address=79.127.219.38:4009 , piece_id=SOTPCWYRPJGDVFUVIWGE33P2S67JV3YDT5P4OCGBLDE5TPH2YHUA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.976671258 (1h23m23.757604643s) [noise=true , address=104.36.82.40:4972 , piece_id=AGCNOB7SKHD44T6UYXGUMS7SNHXN4ZXT4FLX7JAHHZT4NJVWOCTQ, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.977024738 (1h23m23.757958118s) [noise=true , address=104.36.82.47:4529 , piece_id=QAOO3X4KVUGRKO5VF2LBPPD7GWRGIDZMVGZP4VOH4AFAOUYCYMPQ, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.977407411 (1h23m23.758340794s) [noise=true , address=156.146.43.225:4449 , piece_id=HJQLMMF3TYUDVZUB2B5LWPQIANUJH5WENRXPBPU3MEQMYUPWA45Q, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.977769767 (1h23m23.758703159s) [noise=true , address=104.36.82.5:4857 , piece_id=OEZDBT2CC2R5FSPRPMJBBI5WBJWRI4TPENARZ4LSFUHSRN4I26BQ, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.981218882 (1h23m23.762152252s) [noise=true , address=77.90.31.10:4537 , piece_id=AXDEUERUDLN5BHH5SUHPG63LZ2J6L4EAU4UYB53OFFPWWN3VAIIA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:08.049896429 (1h23m23.830829797s) [upload=1287]: Next returned nil segment Jul 26 23:15:08.049917676 (1h23m23.830851061s) [upload=1287]: Waiting for error group managing segments... Jul 26 23:15:08.085192460 (1h23m23.866125836s) [noise=true , address=79.127.219.38:4009 , piece_id=SOTPCWYRPJGDVFUVIWGE33P2S67JV3YDT5P4OCGBLDE5TPH2YHUA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:08.603450848 (1h23m24.384384234s) [noise=true , address=79.127.205.246:4284 , piece_id=G3O7BHQJSM5OL6VKT5QOAF4VENG7IMOUPWQ5THV7MZHRONBDMJCA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:08.665125001 (1h23m24.44605838s) [noise=true , address=79.127.205.235:4686 , piece_id=FQFJS5F3BZKRYMMY7DE7OJSIVMDG4HPQJLNOXYKWSZAMMSWJMRFA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:09.353269591 (1h23m25.134202959s) [noise=true , address=45.91.116.9:4273 , piece_id=JVNX4GTSC54GRKPZVFLLQHC2KDZRC47KBP6BXQDMAL37K5XEIGGA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:10.898271886 (1h23m26.679205253s) [noise=true , address=64.49.13.140:4508 , piece_id=EXU4MEC7557C6G6F3LWBJC4PYVNH263E3AMCXMV6POQK22DRMTYQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:10.931946288 (1h23m26.712879668s) [noise=true , address=77.90.31.15:4713 , piece_id=B6BKAO2PCS2APNPYT2PNRFOFBDJ362YMGFTJNZ66JSNTIAPF6ZNA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:12.100374758 (1h23m27.881308138s) [noise=true , address=156.146.43.225:4449 , piece_id=HJQLMMF3TYUDVZUB2B5LWPQIANUJH5WENRXPBPU3MEQMYUPWA45Q, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:14.060230501 (1h23m29.841163881s) [noise=true , address=45.91.116.11:4121 , piece_id=72UBINQFQAAAB5DJNRNHSNXDUZTN75NLIGWO2XQLLEXQUZFREZEA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:20.405371490 (1h23m36.18630487s) [noise=true , address=77.90.31.19:4860 , piece_id=WOUJICTIKWRWXIX4NWI3DQ52J7EBOZRG3YGFTKZBUR4PQGRU7KCQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:16:07.886099848 (1h24m23.667033231s) [noise=true , address=104.36.82.46:4506 , piece_id=67NG5XPCAIIYY53GEKDCCYRRJTQ3LDPELSFY2PXJT54A2IJCOEOA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: rpc: dial tcp 104.36.82.46:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886173501 (1h24m23.667106878s) [noise=true , address=104.36.82.53:4122 , piece_id=FEUI7GYJZLYQWLTPF2MGUQGHU64XUIDVR3CS6K5LOIBQ75A3URYA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: rpc: dial tcp 104.36.82.53:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886251860 (1h24m23.667185238s) [noise=true , address=104.36.82.55:4206 , piece_id=Q6I5CAH7S6CISDOY3EWWLZTXDISMLKJ3KUQFW2OXAS5GGF2D43BQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:12Mp4ZzYgrP3ZSXDeHbSTXaAJMPgGKUqVjdNtyfeKE5qBMAzBke, address:104.36.82.55:4206): protocol: rpc: dial tcp 104.36.82.55:4206: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886312734 (1h24m23.667246111s) [noise=true , address=104.36.82.40:4972 , piece_id=AGCNOB7SKHD44T6UYXGUMS7SNHXN4ZXT4FLX7JAHHZT4NJVWOCTQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:1XSuvCvq716GAa9WWmM3n2tqnfh8paaFUhjBzVzpiqwJtNifnr, address:104.36.82.40:4972): protocol: rpc: dial tcp 104.36.82.40:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886361447 (1h24m23.667294836s) [noise=true , address=104.36.82.52:4090 , piece_id=PNBVLIRI47XNQAAKEFSHUJ72WUXGHNQWVEOYH3PRXVDOULSZOH3Q, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: rpc: dial tcp 104.36.82.52:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886375590 (1h24m23.66730897s) [noise=true , address=104.36.82.5:4857 , piece_id=OEZDBT2CC2R5FSPRPMJBBI5WBJWRI4TPENARZ4LSFUHSRN4I26BQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: write tcp 159.223.143.39:51994->104.36.82.5:4857: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886423555 (1h24m23.667356934s) [noise=true , address=77.90.31.10:4537 , piece_id=AXDEUERUDLN5BHH5SUHPG63LZ2J6L4EAU4UYB53OFFPWWN3VAIIA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: rpc: dial tcp 77.90.31.10:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886459961 (1h24m23.667393328s) [noise=true , address=104.36.82.47:4529 , piece_id=QAOO3X4KVUGRKO5VF2LBPPD7GWRGIDZMVGZP4VOH4AFAOUYCYMPQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:1m7FrQEps7d8enRYk4v17UxcDs8o8L2CFcVEyB3zR4EniXrYeH, address:104.36.82.47:4529): protocol: rpc: dial tcp 104.36.82.47:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886474039 (1h24m23.66740741s) [noise=true , address=104.36.82.11:4121 , piece_id=HUHD7H3OQD2IK43YMUVDEFXFXZAFP77FRFRBEF6IGX2OBZFEFYHA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:1Fq1JNvrZLTFTQLVF8jzyavJ6dU1xX96FTGcyxfaHiPdCLu4pB, address:104.36.82.11:4121): protocol: rpc: dial tcp 104.36.82.11:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886540011 (1h24m23.66747341s) [seg_pos={0 0}, upload=1287]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:16:07.913708503 (1h24m23.694641871s) [upload=1287]: Done uploading segments... Jul 26 23:16:07.913739321 (1h24m23.694672705s) [upload=1287]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.030293782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X...XX.................X.X..........XX...X...X..XX.X...X..X] duration: 1m0.000715343s FAIL (2025-07-26 23:15:56 UTC) context deadline exceeded Duration: 1m0.000740725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 30.035597826s FAIL (2025-07-26 23:15:49 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 0889088 (1h23m34.911822471s) [upload=1288]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:15:19.169160557 (1h23m34.950093969s) [seg_pos={0 0}, upload=1288]: Begin upload segment... Jul 26 23:15:19.169249223 (1h23m34.950182587s) [seg_pos={0 0}, upload=1288]: Done begin upload segment. Jul 26 23:15:19.169634165 (1h23m34.950567548s) [noise=true , address=77.90.31.14:4661 , piece_id=JI7JV5V7TECV64NNXDUX6T2RRTS7KBE37UYRYEXAK6LOGMWAYDFA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.169704244 (1h23m34.950637631s) [noise=true , address=45.91.116.11:4119 , piece_id=3IGAM3W7O5SXPU27QC673PQA22VI5HSEUMH6I4FZEZ2I5QW6LCIQ, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.170769336 (1h23m34.951702716s) [noise=true , address=104.36.82.15:4259 , piece_id=OZSLW6EVGNHMGXFWY33UYCW2DTMHJXOODQUOE6V55T4YA7DA2B3Q, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.171411757 (1h23m34.952345132s) [noise=true , address=45.91.116.12:4225 , piece_id=EIVGYLUO6DPC53OR3BOBB5SJEFURILWY5I57NTPMBKKDNPZQHTDA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.171553033 (1h23m34.952486425s) [noise=true , address=104.36.82.10:4077 , piece_id=YMSD2J5BSTZMEMTPWISIWXT5YONS3T5XZXDCGDFQCU4LX5I64SDA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.171803754 (1h23m34.952737147s) [noise=true , address=45.91.116.13:4154 , piece_id=YPMXNQ5QNAUGSUVCOA3TJM6CEW5FFVWDPY6QOTNC2GEXAPHDX75Q, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.172387449 (1h23m34.953320831s) [noise=true , address=104.36.82.6:4873 , piece_id=PYC5NUNYC3YUDWXK6IVCYXBX4DCGIRTYDK2IKP32T5DAZ2S4KTSQ, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.172405782 (1h23m34.953339168s) [noise=true , address=104.36.82.4:4804 , piece_id=I7ORPDGOPP3OO4RVG2UVK6ZY62CIHQFQFYUEOFOWBEKS2NF7QC5Q, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.172769369 (1h23m34.953702747s) [noise=true , address=45.91.116.4:4937 , piece_id=Q37QXZPFJALZJLK5LZFKRSL44HQOEMVCVDSWQMLCKBW7PAAJBPCA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.172827575 (1h23m34.953760958s) [noise=true , address=104.36.82.15:4258 , piece_id=7TQ34UE2ZQNQLGWKXF53JYFZUHIFJXFHSFAL5WSR6TWJS6D2TFZQ, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.173153487 (1h23m34.954086864s) [noise=true , address=104.36.82.12:4151 , piece_id=6NKLBOLCAWPZBFILJV7KA66GBYFMBV3B5SJYVVU5JFLUCEKU6OVA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.173195876 (1h23m34.954129264s) [noise=true , address=45.91.116.7:4009 , piece_id=YEHCEEUL74BXL3NGUWX4UCC7KADOR4YJUBWN6TFKKDMTGSMGR6MA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.173622931 (1h23m34.954556315s) [noise=true , address=104.36.82.8:4969 , piece_id=E52UAB65ZSA42EC27J2FRIM5CZUNQJFMM6A623QI7AFGEUZEBFZA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.174555913 (1h23m34.95548929s) [noise=true , address=104.36.82.14:4223 , piece_id=XZ4HBMPQNMK2REAS3Y5IIXUBZLR7V2AY3PEEPS442XS5QPNOLDVQ, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.174955719 (1h23m34.955889101s) [noise=true , address=104.36.82.9:4005 , piece_id=PJIBVQIQV6BYSBDL2I2LAUJQWYS7VSZ2AM4RMGRKP6UATINW6PWA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.176640932 (1h23m34.957574314s) [noise=true , address=104.36.82.10:4078 , piece_id=2O25GYTYH7LYPGBVAZDDOKP5LQDCX4JWAZLYORUYWDROL4CUVUHQ, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.267372139 (1h23m35.048305503s) [upload=1288]: Next returned nil segment Jul 26 23:15:19.267417139 (1h23m35.048350519s) [upload=1288]: Waiting for error group managing segments... Jul 26 23:15:19.272676888 (1h23m35.053610258s) [noise=true , address=45.91.116.4:4937 , piece_id=Q37QXZPFJALZJLK5LZFKRSL44HQOEMVCVDSWQMLCKBW7PAAJBPCA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:19.272742924 (1h23m35.053676295s) [noise=true , address=45.91.116.13:4154 , piece_id=YPMXNQ5QNAUGSUVCOA3TJM6CEW5FFVWDPY6QOTNC2GEXAPHDX75Q, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:19.273195386 (1h23m35.054128756s) [noise=true , address=45.91.116.7:4009 , piece_id=YEHCEEUL74BXL3NGUWX4UCC7KADOR4YJUBWN6TFKKDMTGSMGR6MA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:19.273254719 (1h23m35.054188095s) [noise=true , address=45.91.116.12:4225 , piece_id=EIVGYLUO6DPC53OR3BOBB5SJEFURILWY5I57NTPMBKKDNPZQHTDA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:19.273670316 (1h23m35.054603681s) [noise=true , address=77.90.31.14:4661 , piece_id=JI7JV5V7TECV64NNXDUX6T2RRTS7KBE37UYRYEXAK6LOGMWAYDFA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:19.284729689 (1h23m35.065663056s) [noise=true , address=45.91.116.11:4119 , piece_id=3IGAM3W7O5SXPU27QC673PQA22VI5HSEUMH6I4FZEZ2I5QW6LCIQ, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:49.089057876 (1h24m4.86999126s) [noise=true , address=104.36.82.14:4223 , piece_id=XZ4HBMPQNMK2REAS3Y5IIXUBZLR7V2AY3PEEPS442XS5QPNOLDVQ, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089145180 (1h24m4.870078556s) [noise=true , address=104.36.82.9:4005 , piece_id=PJIBVQIQV6BYSBDL2I2LAUJQWYS7VSZ2AM4RMGRKP6UATINW6PWA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089169179 (1h24m4.870102547s) [noise=true , address=104.36.82.10:4077 , piece_id=YMSD2J5BSTZMEMTPWISIWXT5YONS3T5XZXDCGDFQCU4LX5I64SDA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089231899 (1h24m4.870165288s) [noise=true , address=104.36.82.6:4873 , piece_id=PYC5NUNYC3YUDWXK6IVCYXBX4DCGIRTYDK2IKP32T5DAZ2S4KTSQ, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:59834->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089262394 (1h24m4.870195758s) [noise=true , address=104.36.82.15:4258 , piece_id=7TQ34UE2ZQNQLGWKXF53JYFZUHIFJXFHSFAL5WSR6TWJS6D2TFZQ, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089300387 (1h24m4.870233763s) [noise=true , address=104.36.82.4:4804 , piece_id=I7ORPDGOPP3OO4RVG2UVK6ZY62CIHQFQFYUEOFOWBEKS2NF7QC5Q, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089342821 (1h24m4.870276186s) [noise=true , address=104.36.82.8:4969 , piece_id=E52UAB65ZSA42EC27J2FRIM5CZUNQJFMM6A623QI7AFGEUZEBFZA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:57774->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089368407 (1h24m4.870301787s) [noise=true , address=104.36.82.15:4259 , piece_id=OZSLW6EVGNHMGXFWY33UYCW2DTMHJXOODQUOE6V55T4YA7DA2B3Q, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:56908->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089420664 (1h24m4.870354029s) [noise=true , address=104.36.82.12:4151 , piece_id=6NKLBOLCAWPZBFILJV7KA66GBYFMBV3B5SJYVVU5JFLUCEKU6OVA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:42786->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089442128 (1h24m4.870375506s) [noise=true , address=104.36.82.10:4078 , piece_id=2O25GYTYH7LYPGBVAZDDOKP5LQDCX4JWAZLYORUYWDROL4CUVUHQ, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:44946->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089462712 (1h24m4.870396094s) [seg_pos={0 0}, upload=1288]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:15:49.121947666 (1h24m4.902881041s) [upload=1288]: Done uploading segments... Jul 26 23:15:49.121971551 (1h24m4.902904917s) [upload=1288]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 30.035650025s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................XXX.X....XX.] duration: 16.082839606s FAIL (2025-07-26 23:15:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.082859008s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [X................X........................X......XXXXXXXXXXX] duration: 30.036834536s FAIL (2025-07-26 23:15:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.036853296s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXX] duration: 1m0.031769906s FAIL (2025-07-26 23:15:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== address=104.36.82.54:4168 , piece_id=JS6DI5ATP522ZRWQXBJGVQJGEGNYTDAHFKVSQROHVBTAPIJGWJ2A, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.922621587 (1h22m23.703554999s) [noise=true , address=79.127.219.38:4018 , piece_id=ZZT3ZDSQVPL563RZHOESOXD3CNUOIZOTGZPX6U2C7VX3YMQVSLNA, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.922857781 (1h22m23.703791165s) [noise=true , address=77.90.31.18:4809 , piece_id=RZF7RMGOWWYPC3VE7QAOQT6JA7GL7GRZ2B7VOP3TAIEAI5ZEKPNA, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923220380 (1h22m23.70415377s) [noise=true , address=104.36.82.55:4202 , piece_id=HA6EOQICGIRALS4IQJK36GPEUNIYS7CFRG7VOTGZZUZSLC2S74CQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923273184 (1h22m23.704206563s) [noise=true , address=79.127.205.233:4650 , piece_id=33U5VOAFCDTXLGH6I72A23U5VOOJC556APTUUL2K3EUUEMEETISQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923577211 (1h22m23.704510592s) [noise=true , address=77.90.31.15:4714 , piece_id=O3T4MAGNMQKWZ5G4OGPJDMYNPF3YAW6SVBWPCWYL46VKI3A2NOQQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923623953 (1h22m23.704557331s) [noise=true , address=104.36.82.46:4481 , piece_id=67VLDMNQIXMLKQI67VOPISWFMKMQW7WKOYJFGGROSVQ5D3GWQBQQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923856441 (1h22m23.70478982s) [noise=true , address=104.36.82.53:4126 , piece_id=3R4O2LGB5K2LDMCVANNJVUYCGMCLPRENEQEMER5EB6RTURMFDP3A, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923959847 (1h22m23.704893244s) [noise=true , address=104.36.82.47:4541 , piece_id=ZRKXL3BDWQJUKEMKI46W4URJDBCRA6FVTMTDNFE5VLTJLQISSCSQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.924400437 (1h22m23.705333821s) [noise=true , address=104.36.82.56:4249 , piece_id=KZLFAZW5WOGI72KZ4CWTULC2G2TL5UP442IOWITHPBCN2JG6WQOA, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.924549182 (1h22m23.705482563s) [noise=true , address=79.127.219.51:4212 , piece_id=BQA22URZHXDDFYVYTOLOLKXPHFTFQN7EKWARG4UHKY6ZDZGRKK7A, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.924756911 (1h22m23.70569032s) [noise=true , address=45.91.116.14:4206 , piece_id=3G7HFKGNSRME5H4SGHPECVFF2JRA2RPSA2DVE3KOT4F5XI245HNQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.924956049 (1h22m23.705889442s) [noise=true , address=64.49.13.140:4514 , piece_id=XX42UAGNHNQ6OB6S7FYCSAR6PQHNO43VWLCMMGHF6GMH4VN6WI6Q, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.925124183 (1h22m23.706057572s) [noise=true , address=77.90.31.10:4542 , piece_id=YQCGP3FQFOTBVMZTJIRQF3WDXJAZWCNPM6W45XW27XYFNVVPR6DA, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.925506588 (1h22m23.706439976s) [noise=true , address=64.49.13.155:4916 , piece_id=LKKPDPGWLR7XHBS7EYP5RUTDEW3F7XB226WBSRGK4B6LRL7BNA2Q, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.925516725 (1h22m23.7064501s) [noise=true , address=79.127.201.215:4341 , piece_id=CAJ6KHJDXGZ672I4JBR7GIKJONY3WAAMIOVJ7LKESUC33GOJ3OIA, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.926012823 (1h22m23.706946214s) [noise=true , address=104.36.82.39:4736 , piece_id=F7SMVZ2M6Q7LNGJNTEM6N7T4FDPHFY6IEGGGDLRBNMCWABFN2MEQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.997679796 (1h22m23.77861318s) [upload=1271]: Next returned nil segment Jul 26 23:14:07.997702149 (1h22m23.778635539s) [upload=1271]: Waiting for error group managing segments... Jul 26 23:14:08.005146841 (1h22m23.786080204s) [noise=true , address=79.127.219.38:4018 , piece_id=ZZT3ZDSQVPL563RZHOESOXD3CNUOIZOTGZPX6U2C7VX3YMQVSLNA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:08.054275424 (1h22m23.835208794s) [noise=true , address=77.90.31.18:4809 , piece_id=RZF7RMGOWWYPC3VE7QAOQT6JA7GL7GRZ2B7VOP3TAIEAI5ZEKPNA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:08.185283145 (1h22m23.966216514s) [noise=true , address=79.127.201.215:4341 , piece_id=CAJ6KHJDXGZ672I4JBR7GIKJONY3WAAMIOVJ7LKESUC33GOJ3OIA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:08.474770242 (1h22m24.255703614s) [noise=true , address=45.91.116.14:4206 , piece_id=3G7HFKGNSRME5H4SGHPECVFF2JRA2RPSA2DVE3KOT4F5XI245HNQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:14:08.540594856 (1h22m24.321528227s) [noise=true , address=79.127.205.233:4650 , piece_id=33U5VOAFCDTXLGH6I72A23U5VOOJC556APTUUL2K3EUUEMEETISQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:08.545358834 (1h22m24.326292232s) [noise=true , address=79.127.205.235:4686 , piece_id=GAFVRN4QFOQW2LHMEHY4DS5RXTANVE72HZAS4YXBPCENWGWPXWVA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:09.429808929 (1h22m25.21074229s) [noise=true , address=64.49.13.155:4916 , piece_id=LKKPDPGWLR7XHBS7EYP5RUTDEW3F7XB226WBSRGK4B6LRL7BNA2Q, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:10.290889670 (1h22m26.071823045s) [noise=true , address=77.90.31.15:4714 , piece_id=O3T4MAGNMQKWZ5G4OGPJDMYNPF3YAW6SVBWPCWYL46VKI3A2NOQQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:10.983871167 (1h22m26.764804536s) [noise=true , address=64.49.13.140:4514 , piece_id=XX42UAGNHNQ6OB6S7FYCSAR6PQHNO43VWLCMMGHF6GMH4VN6WI6Q, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:11.096762579 (1h22m26.877695947s) [noise=true , address=79.127.219.51:4212 , piece_id=BQA22URZHXDDFYVYTOLOLKXPHFTFQN7EKWARG4UHKY6ZDZGRKK7A, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:15:07.791380082 (1h23m23.572313464s) [noise=true , address=104.36.82.46:4481 , piece_id=67VLDMNQIXMLKQI67VOPISWFMKMQW7WKOYJFGGROSVQ5D3GWQBQQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791476152 (1h23m23.57240955s) [noise=true , address=104.36.82.53:4126 , piece_id=3R4O2LGB5K2LDMCVANNJVUYCGMCLPRENEQEMER5EB6RTURMFDP3A, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: write tcp 159.223.143.39:35952->104.36.82.53:4126: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791553490 (1h23m23.572486869s) [noise=true , address=77.90.31.10:4542 , piece_id=YQCGP3FQFOTBVMZTJIRQF3WDXJAZWCNPM6W45XW27XYFNVVPR6DA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: write tcp 159.223.143.39:54566->77.90.31.10:4542: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791579072 (1h23m23.572512435s) [noise=true , address=104.36.82.56:4249 , piece_id=KZLFAZW5WOGI72KZ4CWTULC2G2TL5UP442IOWITHPBCN2JG6WQOA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: write tcp 159.223.143.39:54678->104.36.82.56:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791646019 (1h23m23.572579381s) [noise=true , address=104.36.82.54:4168 , piece_id=JS6DI5ATP522ZRWQXBJGVQJGEGNYTDAHFKVSQROHVBTAPIJGWJ2A, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: write tcp 159.223.143.39:59834->104.36.82.54:4168: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791681935 (1h23m23.572615316s) [noise=true , address=104.36.82.39:4736 , piece_id=F7SMVZ2M6Q7LNGJNTEM6N7T4FDPHFY6IEGGGDLRBNMCWABFN2MEQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:126MWCnfveMZSncssS6wf4HGCi4eXvL7FvMD31oYoiR3SEFdGxX, address:104.36.82.39:4736): protocol: write tcp 159.223.143.39:33354->104.36.82.39:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791697549 (1h23m23.572630927s) [noise=true , address=104.36.82.55:4202 , piece_id=HA6EOQICGIRALS4IQJK36GPEUNIYS7CFRG7VOTGZZUZSLC2S74CQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: rpc: dial tcp 104.36.82.55:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791708847 (1h23m23.57264222s) [noise=true , address=104.36.82.47:4541 , piece_id=ZRKXL3BDWQJUKEMKI46W4URJDBCRA6FVTMTDNFE5VLTJLQISSCSQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: rpc: dial tcp 104.36.82.47:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791731505 (1h23m23.572664889s) [seg_pos={0 0}, upload=1271]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:15:07.819695565 (1h23m23.600628941s) [upload=1271]: Done uploading segments... Jul 26 23:15:07.819725538 (1h23m23.600658917s) [upload=1271]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.031794623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X................X........................X......XXXXXXXXXX] duration: 30.110176025s FAIL (2025-07-26 23:14:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.110203956s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................................XXX.X....XX] duration: 30.000612648s FAIL (2025-07-26 23:14:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000636428s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 30.027171395s FAIL (2025-07-26 23:14:36 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== g upload Jul 26 23:14:06.294674324 (1h22m22.075607705s) [upload=1270]: Done starting upload Jul 26 23:14:06.294951447 (1h22m22.075884817s) [upload=1270]: Uploading segments... Jul 26 23:14:06.295004832 (1h22m22.075938201s) [upload=1270]: Got next segment. Inline: false Jul 26 23:14:06.295097519 (1h22m22.076030898s) [upload=1270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:14:06.295103800 (1h22m22.076037177s) [upload=1270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:14:06.344934434 (1h22m22.125867808s) [seg_pos={0 0}, upload=1270]: Begin upload segment... Jul 26 23:14:06.344974552 (1h22m22.125907929s) [seg_pos={0 0}, upload=1270]: Done begin upload segment. Jul 26 23:14:06.344997847 (1h22m22.125931225s) [noise=true , address=104.36.82.11:4114 , piece_id=AFK76IEWKSPYQ426NQWPCLGYBAQ45P5SELOO4ASFCT6J6KEV32QQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.345583583 (1h22m22.12651697s) [noise=true , address=104.36.82.15:4259 , piece_id=I6DGUKCEOHCHLQHEKUU4CEQMSQ6KPKFZQKBNLQLSIQFO423IJIFA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.345790604 (1h22m22.12672399s) [noise=true , address=104.36.82.2:4730 , piece_id=ASDPUQNKNCQ753DOKKG6KC5CL4HBUBQ332GIZYHQZC3CUC5Z7SUA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.345967041 (1h22m22.126900429s) [noise=true , address=45.91.116.6:4974 , piece_id=ORGFBSGD424ROHJBGSY2UNPRNFQF6IMKWTE4QKDKUIEB6U4BHHAQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.346408223 (1h22m22.127341602s) [noise=true , address=77.90.31.14:4661 , piece_id=VR32LJN3C24CL6TZWB4SPF2QPGD73P65MXQNTANHA4E7UENNRO5A, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.346431140 (1h22m22.127364527s) [noise=true , address=45.91.116.5:4297 , piece_id=ATTQE6BLIF362ZKFLCBVJLWBHQD75GCOA2MIHLDG64MVOCZ7NOJQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.346844465 (1h22m22.127777843s) [noise=true , address=45.91.116.13:4155 , piece_id=XD7TPXEFNFGAI7MVQJXNIR3NG4HOCD7YAN4NMX6YSTNOG5VQEZEQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.346901249 (1h22m22.127834631s) [noise=true , address=104.36.82.8:4972 , piece_id=XXLEGIYETDZ2W6SU2AEWXIRN5VLVJW46KJBATRKSXBNJFVA6OEWA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.347365484 (1h22m22.128298868s) [noise=true , address=77.90.31.13:4626 , piece_id=U5VKQOPZKD62DL74PI7AN77OIKNJXA7FFQJ5RVAF73LBWDDOW7SA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.347460108 (1h22m22.128393489s) [noise=true , address=104.36.82.6:4873 , piece_id=MKFGJEZL5CB4SKBZLR5TV3VBHUKWIT7LLZY6I7PDUQJJV43SNJBQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.347779917 (1h22m22.128713292s) [noise=true , address=104.36.82.9:4006 , piece_id=QHZMOJPJZ265XXVY22SPXYBIDXCELFXH6S2Q3GLH3OOGE2MLZLPA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.348226301 (1h22m22.129159687s) [noise=true , address=45.91.116.8:4045 , piece_id=7XVKU2G2D44SDJGESGG3PJ3VG76RFBKOQGO5CEAPLXMGLZBHWZZA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.348707235 (1h22m22.129640613s) [noise=true , address=104.36.82.16:4295 , piece_id=ZCMU5GHJBSOUAAVTQXBXQSTARONRVQLMJDBJAIEAW6PQFKPARFLQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.349051019 (1h22m22.129984396s) [noise=true , address=45.91.116.7:4010 , piece_id=2TM2PQZEXY7L5C7W2ZSNSGJ6LAPTKJ4ECWTVPC3K6TOE4CUCGGLQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.349476364 (1h22m22.130409746s) [noise=true , address=104.36.82.4:4802 , piece_id=OVZAFKRPT354OEXFCTHZIGR2P5QY7GKZD4GNKXFAMDF4ZISFFK5A, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.349844580 (1h22m22.130777966s) [noise=true , address=104.36.82.14:4223 , piece_id=X35LXUFSX3GN355VTBJTIXJPAEG73YCVVSRWMNSROVYHDVFXSLHA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.444969814 (1h22m22.225903183s) [upload=1270]: Next returned nil segment Jul 26 23:14:06.444988096 (1h22m22.225921473s) [upload=1270]: Waiting for error group managing segments... Jul 26 23:14:06.451105122 (1h22m22.232038506s) [noise=true , address=77.90.31.13:4626 , piece_id=U5VKQOPZKD62DL74PI7AN77OIKNJXA7FFQJ5RVAF73LBWDDOW7SA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.451292045 (1h22m22.232225418s) [noise=true , address=45.91.116.5:4297 , piece_id=ATTQE6BLIF362ZKFLCBVJLWBHQD75GCOA2MIHLDG64MVOCZ7NOJQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.451795566 (1h22m22.232728937s) [noise=true , address=45.91.116.7:4010 , piece_id=2TM2PQZEXY7L5C7W2ZSNSGJ6LAPTKJ4ECWTVPC3K6TOE4CUCGGLQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.452269060 (1h22m22.233202437s) [noise=true , address=45.91.116.13:4155 , piece_id=XD7TPXEFNFGAI7MVQJXNIR3NG4HOCD7YAN4NMX6YSTNOG5VQEZEQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.452344082 (1h22m22.23327746s) [noise=true , address=45.91.116.8:4045 , piece_id=7XVKU2G2D44SDJGESGG3PJ3VG76RFBKOQGO5CEAPLXMGLZBHWZZA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.452541627 (1h22m22.233475006s) [noise=true , address=45.91.116.6:4974 , piece_id=ORGFBSGD424ROHJBGSY2UNPRNFQF6IMKWTE4QKDKUIEB6U4BHHAQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.458749432 (1h22m22.239682809s) [noise=true , address=77.90.31.14:4661 , piece_id=VR32LJN3C24CL6TZWB4SPF2QPGD73P65MXQNTANHA4E7UENNRO5A, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.473611380 (1h22m22.25454476s) [noise=true , address=104.36.82.8:4972 , piece_id=XXLEGIYETDZ2W6SU2AEWXIRN5VLVJW46KJBATRKSXBNJFVA6OEWA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:36.263512563 (1h22m52.044445961s) [noise=true , address=104.36.82.4:4802 , piece_id=OVZAFKRPT354OEXFCTHZIGR2P5QY7GKZD4GNKXFAMDF4ZISFFK5A, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.263626431 (1h22m52.044559808s) [noise=true , address=104.36.82.2:4730 , piece_id=ASDPUQNKNCQ753DOKKG6KC5CL4HBUBQ332GIZYHQZC3CUC5Z7SUA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.263694596 (1h22m52.044627974s) [noise=true , address=104.36.82.15:4259 , piece_id=I6DGUKCEOHCHLQHEKUU4CEQMSQ6KPKFZQKBNLQLSIQFO423IJIFA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:56564->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.263831405 (1h22m52.044764791s) [noise=true , address=104.36.82.9:4006 , piece_id=QHZMOJPJZ265XXVY22SPXYBIDXCELFXH6S2Q3GLH3OOGE2MLZLPA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:47842->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.263916310 (1h22m52.044849688s) [noise=true , address=104.36.82.14:4223 , piece_id=X35LXUFSX3GN355VTBJTIXJPAEG73YCVVSRWMNSROVYHDVFXSLHA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:44766->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.264105934 (1h22m52.04503952s) [noise=true , address=104.36.82.11:4114 , piece_id=AFK76IEWKSPYQ426NQWPCLGYBAQ45P5SELOO4ASFCT6J6KEV32QQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:34796->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.264243249 (1h22m52.045176627s) [noise=true , address=104.36.82.6:4873 , piece_id=MKFGJEZL5CB4SKBZLR5TV3VBHUKWIT7LLZY6I7PDUQJJV43SNJBQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:36958->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.264252640 (1h22m52.045186019s) [noise=true , address=104.36.82.16:4295 , piece_id=ZCMU5GHJBSOUAAVTQXBXQSTARONRVQLMJDBJAIEAW6PQFKPARFLQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:39936->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.264296199 (1h22m52.045229585s) [seg_pos={0 0}, upload=1270]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:14:36.287847086 (1h22m52.06878047s) [upload=1270]: Done uploading segments... Jul 26 23:14:36.287879918 (1h22m52.068813304s) [upload=1270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.027215315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 827.637656ms FAIL (2025-07-26 23:14:36 UTC) orbiter warning: uplink: too many requests Duration: 827.676978ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X................X........................X......XXXXXXXXX] duration: 30.442718329s FAIL (2025-07-26 23:14:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.442736603s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXX] duration: 1m0.062656014s FAIL (2025-07-26 23:14:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== NDO4NJS75AFDEUIXFZVXHS5P4I7OXWJ3YGJU3P3UODP3MENAA, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.716665559 (1h21m23.497599059s) [noise=true , address=79.127.219.51:4210 , piece_id=P3WNGV5ZI4EROKLYU52AY437OUIKLWQHFYQXV6UYMIPPL2S2KL2A, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.717302083 (1h21m23.498235472s) [noise=true , address=104.36.82.54:4171 , piece_id=MBIKNFBMDSF4ZJ4OVKAVX6DYBNKEOTOIP4F5REFDUSQY2U3MK2CQ, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.717365182 (1h21m23.49829857s) [noise=true , address=79.127.205.234:4603 , piece_id=NBLBGX4ORKC62UPPGMFVWOYSXVZZGKSX2I2P53MHS5MNOCRC246A, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.717738312 (1h21m23.4986717s) [noise=true , address=79.127.205.233:4650 , piece_id=JACP6H362M2UHKTBXGVOUTSNRUCB32AHE2U754VZW3RG2AMXNRHA, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.718129776 (1h21m23.499063163s) [noise=true , address=79.127.205.245:4242 , piece_id=SHNZFTKZLT7OICCY2KJCIAGFK4IPE7IBDMMGHHRFXQRT356GOGOQ, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.718595332 (1h21m23.499528722s) [noise=true , address=77.90.31.17:4786 , piece_id=MGQJPHHGLCKPT5GU4J355C6ISNBBKPVTTU4I72UPMHLAPZIJYOPQ, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.718877103 (1h21m23.499810488s) [noise=true , address=77.90.31.15:4713 , piece_id=B56GNLNLBEADLWBSX52BO26KVYILHNLCAUF6XDIIHVAMPWH5T7VA, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.719376317 (1h21m23.500309704s) [noise=true , address=77.90.31.11:4565 , piece_id=YP53SBHA3RHDGPV5NXUDUEARZPHM2NWR2MHEGMZUNTJDTW4YDBEQ, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.719727255 (1h21m23.500660634s) [noise=true , address=104.36.82.53:4146 , piece_id=KK55HWZ7AFVFWVYNVKYIMYP6OSNZSP7HQ2CLUXP7XHWPQSVAZHMA, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.720113734 (1h21m23.501047124s) [noise=true , address=104.36.82.20:4465 , piece_id=ECGJXJU2EZ4BYDIYSOICSNZVLDQQN6ILKBWVWMFFWIUAZMOQFPYA, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.720493533 (1h21m23.501426919s) [noise=true , address=104.36.82.56:4238 , piece_id=5SEH5AVFKCZWIS4OSJEAETTBDHDYQIIBRCS6GWVK55JY223LWNGQ, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.720705895 (1h21m23.501639396s) [noise=true , address=64.49.13.138:4483 , piece_id=XH4HTCIFBTB4VGU4PZFKMNL4QNJGOP7RMO6LISONCTHPCTD5YQRQ, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.721238163 (1h21m23.50217155s) [noise=true , address=45.91.116.5:4300 , piece_id=P7KAMCAJRBOMGR3SGK4VFBJYRUWUMPHSJJKWZOYPCIJWGCU6GYZA, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.721748864 (1h21m23.502682252s) [noise=true , address=104.36.82.55:4197 , piece_id=QNCM3RF2JCNGPQTJA6BJZYB37IZIIVO6ZMPXE3VC4CUY5LYZVY4A, seg_pos={0 0}, upload=1254]: Uploading piece... Jul 26 23:13:07.758079171 (1h21m23.539012542s) [noise=true , address=45.91.116.5:4300 , piece_id=P7KAMCAJRBOMGR3SGK4VFBJYRUWUMPHSJJKWZOYPCIJWGCU6GYZA, seg_pos={0 0}, upload=1254]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: not enough available disk space, have: 0, need: 6725632 Jul 26 23:13:07.799696508 (1h21m23.580629886s) [upload=1254]: Next returned nil segment Jul 26 23:13:07.799795727 (1h21m23.580729112s) [upload=1254]: Waiting for error group managing segments... Jul 26 23:13:07.829872540 (1h21m23.610805905s) [noise=true , address=79.127.219.38:4003 , piece_id=MMGYTJBQCCRYDMTVZRFXTOK26PHTOUXTT4GXS2WVJWHZT2ZX5NBQ, seg_pos={0 0}, upload=1254]: Done uploading piece. err: <nil> Jul 26 23:13:07.832818780 (1h21m23.613752159s) [noise=true , address=77.90.31.11:4565 , piece_id=YP53SBHA3RHDGPV5NXUDUEARZPHM2NWR2MHEGMZUNTJDTW4YDBEQ, seg_pos={0 0}, upload=1254]: Done uploading piece. err: <nil> Jul 26 23:13:08.312302185 (1h21m24.093235562s) [noise=true , address=79.127.205.234:4603 , piece_id=NBLBGX4ORKC62UPPGMFVWOYSXVZZGKSX2I2P53MHS5MNOCRC246A, seg_pos={0 0}, upload=1254]: Done uploading piece. err: <nil> Jul 26 23:13:08.334332006 (1h21m24.115265384s) [noise=true , address=79.127.205.233:4650 , piece_id=JACP6H362M2UHKTBXGVOUTSNRUCB32AHE2U754VZW3RG2AMXNRHA, seg_pos={0 0}, upload=1254]: Done uploading piece. err: <nil> Jul 26 23:13:08.362097890 (1h21m24.143031269s) [noise=true , address=79.127.205.245:4242 , piece_id=SHNZFTKZLT7OICCY2KJCIAGFK4IPE7IBDMMGHHRFXQRT356GOGOQ, seg_pos={0 0}, upload=1254]: Done uploading piece. err: <nil> Jul 26 23:13:09.758427620 (1h21m25.539360982s) [noise=true , address=79.127.219.51:4210 , piece_id=P3WNGV5ZI4EROKLYU52AY437OUIKLWQHFYQXV6UYMIPPL2S2KL2A, seg_pos={0 0}, upload=1254]: Done uploading piece. err: <nil> Jul 26 23:13:09.885524679 (1h21m25.666458047s) [noise=true , address=64.49.13.138:4483 , piece_id=XH4HTCIFBTB4VGU4PZFKMNL4QNJGOP7RMO6LISONCTHPCTD5YQRQ, seg_pos={0 0}, upload=1254]: Done uploading piece. err: <nil> Jul 26 23:13:11.834244152 (1h21m27.615177515s) [noise=true , address=77.90.31.15:4713 , piece_id=B56GNLNLBEADLWBSX52BO26KVYILHNLCAUF6XDIIHVAMPWH5T7VA, seg_pos={0 0}, upload=1254]: Done uploading piece. err: <nil> Jul 26 23:14:07.638932037 (1h22m23.419865415s) [noise=true , address=104.36.82.53:4146 , piece_id=KK55HWZ7AFVFWVYNVKYIMYP6OSNZSP7HQ2CLUXP7XHWPQSVAZHMA, seg_pos={0 0}, upload=1254]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 159.223.143.39:37964->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:07.639004515 (1h22m23.419937891s) [noise=true , address=104.36.82.46:4491 , piece_id=HCVNDO4NJS75AFDEUIXFZVXHS5P4I7OXWJ3YGJU3P3UODP3MENAA, seg_pos={0 0}, upload=1254]: Done uploading piece. err: ecclient: upload failed (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX, address:104.36.82.46:4491): protocol: rpc: dial tcp 104.36.82.46:4491: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:07.639093092 (1h22m23.420026468s) [noise=true , address=104.36.82.47:4522 , piece_id=HWSNL4MYKF2N7IYI6JFKFRSZAAI5DATIF3N6VNAARYKSQEFBMCRQ, seg_pos={0 0}, upload=1254]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: write tcp 159.223.143.39:41320->104.36.82.47:4522: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:07.639151697 (1h22m23.420085073s) [noise=true , address=104.36.82.54:4171 , piece_id=MBIKNFBMDSF4ZJ4OVKAVX6DYBNKEOTOIP4F5REFDUSQY2U3MK2CQ, seg_pos={0 0}, upload=1254]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: write tcp 159.223.143.39:41142->104.36.82.54:4171: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:07.639200518 (1h22m23.420133885s) [noise=true , address=104.36.82.55:4197 , piece_id=QNCM3RF2JCNGPQTJA6BJZYB37IZIIVO6ZMPXE3VC4CUY5LYZVY4A, seg_pos={0 0}, upload=1254]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: write tcp 159.223.143.39:60130->104.36.82.55:4197: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:07.639241393 (1h22m23.42017476s) [noise=true , address=104.36.82.52:4112 , piece_id=JHD47J2QRBB67XDOT7OODMSXBMM4FRFYKFI77LAPRUBMLDUQ6LUQ, seg_pos={0 0}, upload=1254]: Done uploading piece. err: ecclient: upload failed (node:1dQFyMxQ1vVeHWCmyUWFUtJNBTdRUP2bbdLJQCMwF9z1L1eapD, address:104.36.82.52:4112): protocol: write tcp 159.223.143.39:51066->104.36.82.52:4112: use of closed network connection; write tcp 159.223.143.39:51066->104.36.82.52:4112: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51066->104.36.82.52:4112: use of closed network connection Jul 26 23:14:07.639248136 (1h22m23.420181505s) [noise=true , address=104.36.82.20:4465 , piece_id=ECGJXJU2EZ4BYDIYSOICSNZVLDQQN6ILKBWVWMFFWIUAZMOQFPYA, seg_pos={0 0}, upload=1254]: Done uploading piece. err: ecclient: upload failed (node:1aGm17yvKPgD3nqcQyoMVvgM3gqXDDrKPNzXJY1xf2uBdcfobS, address:104.36.82.20:4465): protocol: write tcp 159.223.143.39:36262->104.36.82.20:4465: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:07.639258352 (1h22m23.42019172s) [noise=true , address=104.36.82.56:4238 , piece_id=5SEH5AVFKCZWIS4OSJEAETTBDHDYQIIBRCS6GWVK55JY223LWNGQ, seg_pos={0 0}, upload=1254]: Done uploading piece. err: ecclient: upload failed (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5, address:104.36.82.56:4238): protocol: write tcp 159.223.143.39:57934->104.36.82.56:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:07.639266383 (1h22m23.420199744s) [noise=true , address=77.90.31.17:4786 , piece_id=MGQJPHHGLCKPT5GU4J355C6ISNBBKPVTTU4I72UPMHLAPZIJYOPQ, seg_pos={0 0}, upload=1254]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: rpc: dial tcp 77.90.31.17:4786: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:07.639324163 (1h22m23.420257547s) [seg_pos={0 0}, upload=1254]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:14:07.697477766 (1h22m23.478411146s) [upload=1254]: Done uploading segments... Jul 26 23:14:07.697507807 (1h22m23.478441184s) [upload=1254]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.062691598s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X.X...XX.................X.X..........XX...X...X..XX.X...X..] duration: 20.971987723s FAIL (2025-07-26 23:14:07 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.972020059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.899407807s FAIL (2025-07-26 23:14:02 UTC) orbiter warning: uplink: too many requests Duration: 2.899453276s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XXX.X....X] duration: 30.000705073s FAIL (2025-07-26 23:13:55 UTC) orbiter warning: context deadline exceeded Duration: 30.000733615s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X................X........................X......XXXXXXXX] duration: 30.270890319s FAIL (2025-07-26 23:13:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.270928128s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.646938927s FAIL (2025-07-26 23:13:39 UTC) orbiter warning: uplink: too many requests Duration: 1.646968475s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X...................................................XXXXXXXX] duration: 30.038813904s FAIL (2025-07-26 23:13:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ad=1238]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:12:29.255589127 (1h20m45.036522523s) [upload=1238]: Done uploading segments... Jul 26 23:12:29.255618348 (1h20m45.036551714s) [upload=1238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:12:57.321454502 (1h21m13.102387866s) [upload=1252]: Starting upload Jul 26 23:12:57.321545256 (1h21m13.102478623s) [upload=1252]: Done starting upload Jul 26 23:12:57.321684295 (1h21m13.10261767s) [upload=1252]: Uploading segments... Jul 26 23:12:57.321902345 (1h21m13.102835712s) [upload=1252]: Got next segment. Inline: false Jul 26 23:12:57.321974045 (1h21m13.102907439s) [upload=1252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:12:57.321981548 (1h21m13.102914935s) [upload=1252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:12:57.352282660 (1h21m13.133216048s) [seg_pos={0 0}, upload=1252]: Begin upload segment... Jul 26 23:12:57.352335262 (1h21m13.133268639s) [seg_pos={0 0}, upload=1252]: Done begin upload segment. Jul 26 23:12:57.352365836 (1h21m13.133299221s) [noise=true , address=45.91.116.6:4975 , piece_id=OIZR7TDGZ5VWDTP2YVIIQMEIUKPI2SVZQW6VBTPE46LU4ANYOAWA, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.352907553 (1h21m13.13384094s) [noise=true , address=45.91.116.10:4081 , piece_id=CZ45GF5KHRJEPVJQLW76P3UBX5MEIUCJBPSQSWNNS4U4IPUDUBNQ, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.353318637 (1h21m13.134252024s) [noise=true , address=45.91.116.5:4297 , piece_id=N7ZX3MWFZUBIWUHNTKJEX3HJ6PLJ46INOU6H2AFVBJSEAH7VLEQQ, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.353754351 (1h21m13.134687721s) [noise=true , address=104.36.82.9:4005 , piece_id=OTEWX73DTDQ7IRZ72GPFP4UBJ5RDJZS7GULCYEAVOY7NUGTBYVBA, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.354133995 (1h21m13.135067382s) [noise=true , address=104.36.82.4:4804 , piece_id=IXGYXBWAQONXQG3PTMGJL3STHXJHYKPJSVZNBU5DTOVMAK5YA4LA, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.354654876 (1h21m13.13558826s) [noise=true , address=104.36.82.6:4873 , piece_id=GQEN4TLN6TYCDHQT74AUSA4MLYYYOHZ4KCOBG7KX6W4YTMHX5HAQ, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.355101068 (1h21m13.136034457s) [noise=true , address=104.36.82.16:4295 , piece_id=53FDVP7C4E4WPW462H6VYQ5O6G7E5EWECICITOIV7JPBFQ3EL6OA, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.355475733 (1h21m13.136409117s) [noise=true , address=104.36.82.15:4257 , piece_id=WYID76FDVESLLVJG2OKGT4OT7U6USBIED644SLIBM4JZPPP6GGWA, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.355841585 (1h21m13.136774972s) [noise=true , address=77.90.31.10:4520 , piece_id=Y6SMUHYDQBIPCPRSTOQD24UIQKTHLSBPYVPRTD7HIKXHTO62SC3Q, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.356285795 (1h21m13.137219179s) [noise=true , address=45.91.116.11:4119 , piece_id=OIJOUUPOQLUG33WU763AG6VEX2C3X7DS37I7BRZUB2VNJ4Y4XGJQ, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.356652966 (1h21m13.13758635s) [noise=true , address=45.91.116.12:4227 , piece_id=BIVDXJ6PDH5SOCFMUYZT5Y3MGK6LNZHS2JHRMHTEQQNJUO4ZZT6Q, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.357055405 (1h21m13.137988786s) [noise=true , address=104.36.82.3:4765 , piece_id=OFUYM6UW23WIU7Y4HD4SJNRMOATUDX7HXMFA4O2QEYNRJN7KPX7Q, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.357431062 (1h21m13.138364445s) [noise=true , address=104.36.82.14:4223 , piece_id=554VCIJQEZKWR5G27YEQQZ7NVEHVSJROEZQKBQXCD74PEPZXO3LQ, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.358021911 (1h21m13.138955296s) [noise=true , address=45.91.116.7:4009 , piece_id=JT64GYD2PIVTIZEB76ATMVNJGFUKDPILCDZSWFZCTWL7GHLOCP2A, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.367103608 (1h21m13.148036994s) [noise=true , address=45.91.116.12:4226 , piece_id=REPIDGPKDV2U7SHOOGMQM6JUNP4JPJGFCCUHWVDTWVYL4O3SN6RQ, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.370640662 (1h21m13.151574047s) [noise=true , address=45.91.116.8:4048 , piece_id=AM4CLXAMNM4PKSZHZVOSRVL4BE3DDVXHQ4Z6GK2UUEBW5CWDHHUA, seg_pos={0 0}, upload=1252]: Uploading piece... Jul 26 23:12:57.494472254 (1h21m13.27540564s) [upload=1252]: Next returned nil segment Jul 26 23:12:57.494499709 (1h21m13.275433098s) [upload=1252]: Waiting for error group managing segments... Jul 26 23:12:57.504506084 (1h21m13.285439467s) [noise=true , address=45.91.116.5:4297 , piece_id=N7ZX3MWFZUBIWUHNTKJEX3HJ6PLJ46INOU6H2AFVBJSEAH7VLEQQ, seg_pos={0 0}, upload=1252]: Done uploading piece. err: <nil> Jul 26 23:12:57.507542144 (1h21m13.288475608s) [noise=true , address=45.91.116.7:4009 , piece_id=JT64GYD2PIVTIZEB76ATMVNJGFUKDPILCDZSWFZCTWL7GHLOCP2A, seg_pos={0 0}, upload=1252]: Done uploading piece. err: <nil> Jul 26 23:12:57.508515526 (1h21m13.289448959s) [noise=true , address=45.91.116.8:4048 , piece_id=AM4CLXAMNM4PKSZHZVOSRVL4BE3DDVXHQ4Z6GK2UUEBW5CWDHHUA, seg_pos={0 0}, upload=1252]: Done uploading piece. err: <nil> Jul 26 23:12:57.509205907 (1h21m13.290139293s) [noise=true , address=45.91.116.6:4975 , piece_id=OIZR7TDGZ5VWDTP2YVIIQMEIUKPI2SVZQW6VBTPE46LU4ANYOAWA, seg_pos={0 0}, upload=1252]: Done uploading piece. err: <nil> Jul 26 23:12:57.510115465 (1h21m13.291048845s) [noise=true , address=45.91.116.12:4226 , piece_id=REPIDGPKDV2U7SHOOGMQM6JUNP4JPJGFCCUHWVDTWVYL4O3SN6RQ, seg_pos={0 0}, upload=1252]: Done uploading piece. err: <nil> Jul 26 23:12:57.511623879 (1h21m13.292557256s) [noise=true , address=45.91.116.12:4227 , piece_id=BIVDXJ6PDH5SOCFMUYZT5Y3MGK6LNZHS2JHRMHTEQQNJUO4ZZT6Q, seg_pos={0 0}, upload=1252]: Done uploading piece. err: <nil> Jul 26 23:12:57.514042595 (1h21m13.294975973s) [noise=true , address=45.91.116.10:4081 , piece_id=CZ45GF5KHRJEPVJQLW76P3UBX5MEIUCJBPSQSWNNS4U4IPUDUBNQ, seg_pos={0 0}, upload=1252]: Done uploading piece. err: <nil> Jul 26 23:12:57.524340208 (1h21m13.305273576s) [noise=true , address=45.91.116.11:4119 , piece_id=OIJOUUPOQLUG33WU763AG6VEX2C3X7DS37I7BRZUB2VNJ4Y4XGJQ, seg_pos={0 0}, upload=1252]: Done uploading piece. err: <nil> Jul 26 23:12:59.783588957 (1h21m15.56452232s) [noise=true , address=77.90.31.10:4520 , piece_id=Y6SMUHYDQBIPCPRSTOQD24UIQKTHLSBPYVPRTD7HIKXHTO62SC3Q, seg_pos={0 0}, upload=1252]: Done uploading piece. err: <nil> Jul 26 23:13:27.290138542 (1h21m43.071071924s) [noise=true , address=104.36.82.16:4295 , piece_id=53FDVP7C4E4WPW462H6VYQ5O6G7E5EWECICITOIV7JPBFQ3EL6OA, seg_pos={0 0}, upload=1252]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:27.290304664 (1h21m43.071238041s) [noise=true , address=104.36.82.14:4223 , piece_id=554VCIJQEZKWR5G27YEQQZ7NVEHVSJROEZQKBQXCD74PEPZXO3LQ, seg_pos={0 0}, upload=1252]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:27.290485010 (1h21m43.071418392s) [noise=true , address=104.36.82.6:4873 , piece_id=GQEN4TLN6TYCDHQT74AUSA4MLYYYOHZ4KCOBG7KX6W4YTMHX5HAQ, seg_pos={0 0}, upload=1252]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:27.290544405 (1h21m43.071477783s) [noise=true , address=104.36.82.4:4804 , piece_id=IXGYXBWAQONXQG3PTMGJL3STHXJHYKPJSVZNBU5DTOVMAK5YA4LA, seg_pos={0 0}, upload=1252]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:41630->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:27.290553776 (1h21m43.071487145s) [noise=true , address=104.36.82.15:4257 , piece_id=WYID76FDVESLLVJG2OKGT4OT7U6USBIED644SLIBM4JZPPP6GGWA, seg_pos={0 0}, upload=1252]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:27.290602085 (1h21m43.07153545s) [noise=true , address=104.36.82.3:4765 , piece_id=OFUYM6UW23WIU7Y4HD4SJNRMOATUDX7HXMFA4O2QEYNRJN7KPX7Q, seg_pos={0 0}, upload=1252]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:27.290608439 (1h21m43.071541806s) [noise=true , address=104.36.82.9:4005 , piece_id=OTEWX73DTDQ7IRZ72GPFP4UBJ5RDJZS7GULCYEAVOY7NUGTBYVBA, seg_pos={0 0}, upload=1252]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:43584->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:13:27.290644303 (1h21m43.071577689s) [seg_pos={0 0}, upload=1252]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:13:27.327363740 (1h21m43.108297109s) [upload=1252]: Done uploading segments... Jul 26 23:13:27.327387962 (1h21m43.108321339s) [upload=1252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.038844523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 332.46842ms FAIL (2025-07-26 23:13:21 UTC) orbiter warning: uplink: too many requests Duration: 332.502482ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X.X...XX.................X.X..........XX...X...X..XX.X...X.] duration: 31.981010291s FAIL (2025-07-26 23:13:15 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.981073727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XXX.X....] duration: 6.56773541s FAIL (2025-07-26 23:13:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.567847332s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXX] duration: 1m0.034911223s FAIL (2025-07-26 23:12:54 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== X4MRIFF4OYVNNIMCN2IVNK66S5JGIY3YWLABI5ESBG7XDF2IGMQ, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.065003827 (1h20m9.845937222s) [noise=true , address=104.36.82.56:4240 , piece_id=HUQ2272ZSOGH7AARE76O6N527UD5TQMRL3DCMQQQO6RPNB5IWCIQ, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.066183013 (1h20m9.8471164s) [noise=true , address=77.90.31.19:4850 , piece_id=32CVUFHKEBS6LXL5ZQR2CW6JFDO4BJTS5MNPH2L2OMIXUOH7RO6A, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.066977897 (1h20m9.847911286s) [noise=true , address=104.36.82.60:4361 , piece_id=ALKRNQXI4DBOUYJ7IUFE2S7JO27EZYWFWX673W3ATL4IFPHHDFVQ, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.068480484 (1h20m9.849413874s) [noise=true , address=77.90.31.11:4567 , piece_id=7GJ6GCFWEKHA3WZAFHRGALNCDG57MDHXGUNFJSJ3YTE2YNZBI25Q, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.069455569 (1h20m9.850388959s) [noise=true , address=104.36.82.52:4105 , piece_id=ED77MUHCP6KYXN5ON57QQSYMWZUECT5WYN3YY4POAU45XSUSK2YQ, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.070427861 (1h20m9.851361297s) [noise=true , address=104.36.82.46:4497 , piece_id=435NVCAEVW2JR4JO2T4OZ6FSKF24UZQT25LR7AQOZ4WHVVLWEZUQ, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.071822044 (1h20m9.852755489s) [noise=true , address=104.36.82.53:4133 , piece_id=FQ46TB5XSXDSDGDDVL7DJAN6UBGWM4FJ6IIKF76V2RWRECZZJTAQ, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.073012677 (1h20m9.853946059s) [noise=true , address=104.36.82.47:4552 , piece_id=GU5YG4HBISSSZQVGDWXCIGKZU5J47RQXS2FIRI5R3JTTC2ZUNPDA, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.074040967 (1h20m9.854974347s) [noise=true , address=79.127.205.245:4248 , piece_id=PYZEJDE5R34M33NWWHOS6WCEOJVL2NK7GMSJBCIPQKSMSJLZPNYA, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.075131496 (1h20m9.856064862s) [noise=true , address=79.127.219.47:4069 , piece_id=DO2O7D3JU5ISCKN4TCWIRMKXM5F4DWPUTFQX5AI5IOHKX2L5MRBA, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.076148911 (1h20m9.85708229s) [noise=true , address=79.127.201.215:4343 , piece_id=5G2XJJRZW42J637B3P5AXRUEJ3H4ISCKMX2TYCB2XJ2CBVLFVXBA, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.078860856 (1h20m9.859794239s) [noise=true , address=104.36.82.54:4159 , piece_id=DJZIR4GYNV4OSZKV6FNRAJQNLNQJTSH6AUXDDTLD2XCVBWXDKHYA, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.081577708 (1h20m9.862511094s) [noise=true , address=79.127.219.51:4212 , piece_id=HKG6FZUUN2SJ2BVYC2QKFPIXZZ4U372OKRTG6EOJ6B7I4ULUAXZQ, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.084679930 (1h20m9.865613326s) [noise=true , address=79.127.219.38:4001 , piece_id=QIHWV44QGWSIVDWDN4C7DLS3NY56KTUBPBLTG4X4BGQTEJW44M4Q, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.087407916 (1h20m9.868341304s) [noise=true , address=104.36.82.55:4197 , piece_id=UD2X3RPNVJUM5Q3IZOEQW2UGDYCQSRA7RHVR76NXXMJUACPXJEEA, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.090701643 (1h20m9.871635028s) [noise=true , address=156.146.43.225:4434 , piece_id=DKYGUL77ASAQZOMGDQK6B25Y73INH6MOIFYX2CSJVFED2G7PE3AQ, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.093953832 (1h20m9.874887212s) [noise=true , address=77.90.31.21:4911 , piece_id=QHQXMWNWG3JU7GCSLVB3BRP6UCGH56WYWYC3TA3ZSA7XMKQLMSLA, seg_pos={0 0}, upload=1236]: Uploading piece... Jul 26 23:11:54.157850000 (1h20m9.938783365s) [upload=1236]: Next returned nil segment Jul 26 23:11:54.157866782 (1h20m9.938800147s) [upload=1236]: Waiting for error group managing segments... Jul 26 23:11:54.234702772 (1h20m10.015636167s) [noise=true , address=156.146.43.225:4434 , piece_id=DKYGUL77ASAQZOMGDQK6B25Y73INH6MOIFYX2CSJVFED2G7PE3AQ, seg_pos={0 0}, upload=1236]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 23:11:54.270974400 (1h20m10.051907781s) [noise=true , address=77.90.31.11:4567 , piece_id=7GJ6GCFWEKHA3WZAFHRGALNCDG57MDHXGUNFJSJ3YTE2YNZBI25Q, seg_pos={0 0}, upload=1236]: Done uploading piece. err: <nil> Jul 26 23:11:54.282352501 (1h20m10.063285881s) [noise=true , address=77.90.31.21:4911 , piece_id=QHQXMWNWG3JU7GCSLVB3BRP6UCGH56WYWYC3TA3ZSA7XMKQLMSLA, seg_pos={0 0}, upload=1236]: Done uploading piece. err: <nil> Jul 26 23:11:54.305190903 (1h20m10.086124285s) [noise=true , address=79.127.219.38:4001 , piece_id=QIHWV44QGWSIVDWDN4C7DLS3NY56KTUBPBLTG4X4BGQTEJW44M4Q, seg_pos={0 0}, upload=1236]: Done uploading piece. err: <nil> Jul 26 23:11:54.391811916 (1h20m10.172745284s) [noise=true , address=104.36.82.60:4361 , piece_id=ALKRNQXI4DBOUYJ7IUFE2S7JO27EZYWFWX673W3ATL4IFPHHDFVQ, seg_pos={0 0}, upload=1236]: Done uploading piece. err: <nil> Jul 26 23:11:54.665943084 (1h20m10.446876495s) [noise=true , address=79.127.205.245:4248 , piece_id=PYZEJDE5R34M33NWWHOS6WCEOJVL2NK7GMSJBCIPQKSMSJLZPNYA, seg_pos={0 0}, upload=1236]: Done uploading piece. err: <nil> Jul 26 23:11:55.534639311 (1h20m11.315572673s) [noise=true , address=79.127.219.51:4212 , piece_id=HKG6FZUUN2SJ2BVYC2QKFPIXZZ4U372OKRTG6EOJ6B7I4ULUAXZQ, seg_pos={0 0}, upload=1236]: Done uploading piece. err: <nil> Jul 26 23:11:58.896936187 (1h20m14.677869566s) [noise=true , address=79.127.219.47:4069 , piece_id=DO2O7D3JU5ISCKN4TCWIRMKXM5F4DWPUTFQX5AI5IOHKX2L5MRBA, seg_pos={0 0}, upload=1236]: Done uploading piece. err: <nil> Jul 26 23:12:05.093709885 (1h20m20.874643247s) [noise=true , address=79.127.201.215:4343 , piece_id=5G2XJJRZW42J637B3P5AXRUEJ3H4ISCKMX2TYCB2XJ2CBVLFVXBA, seg_pos={0 0}, upload=1236]: Done uploading piece. err: <nil> Jul 26 23:12:17.416508014 (1h20m33.197441392s) [noise=true , address=77.90.31.16:4736 , piece_id=UX4MRIFF4OYVNNIMCN2IVNK66S5JGIY3YWLABI5ESBG7XDF2IGMQ, seg_pos={0 0}, upload=1236]: Done uploading piece. err: <nil> Jul 26 23:12:28.688305200 (1h20m44.469238563s) [noise=true , address=77.90.31.19:4850 , piece_id=32CVUFHKEBS6LXL5ZQR2CW6JFDO4BJTS5MNPH2L2OMIXUOH7RO6A, seg_pos={0 0}, upload=1236]: Done uploading piece. err: <nil> Jul 26 23:12:53.995939072 (1h21m9.776872447s) [noise=true , address=104.36.82.56:4240 , piece_id=HUQ2272ZSOGH7AARE76O6N527UD5TQMRL3DCMQQQO6RPNB5IWCIQ, seg_pos={0 0}, upload=1236]: Done uploading piece. err: ecclient: upload failed (node:129QqkFEzZdtzQ6HAKxPWxDNgDVUNJJLH59oRVXnHExrwkDECyf, address:104.36.82.56:4240): protocol: write tcp 159.223.143.39:50020->104.36.82.56:4240: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:53.996140667 (1h21m9.777074047s) [noise=true , address=104.36.82.52:4105 , piece_id=ED77MUHCP6KYXN5ON57QQSYMWZUECT5WYN3YY4POAU45XSUSK2YQ, seg_pos={0 0}, upload=1236]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:104.36.82.52:4105): protocol: write tcp 159.223.143.39:44252->104.36.82.52:4105: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:53.996195448 (1h21m9.777128812s) [noise=true , address=104.36.82.46:4497 , piece_id=435NVCAEVW2JR4JO2T4OZ6FSKF24UZQT25LR7AQOZ4WHVVLWEZUQ, seg_pos={0 0}, upload=1236]: Done uploading piece. err: ecclient: upload failed (node:12Zg4PSYBuLxFvF9oYiuvn65JNbjT5RdagcVAQLp2izvjPe1NEc, address:104.36.82.46:4497): protocol: write tcp 159.223.143.39:34018->104.36.82.46:4497: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:53.996405257 (1h21m9.777338641s) [noise=true , address=104.36.82.55:4197 , piece_id=UD2X3RPNVJUM5Q3IZOEQW2UGDYCQSRA7RHVR76NXXMJUACPXJEEA, seg_pos={0 0}, upload=1236]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: write tcp 159.223.143.39:35786->104.36.82.55:4197: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:53.996519589 (1h21m9.777452961s) [noise=true , address=104.36.82.54:4159 , piece_id=DJZIR4GYNV4OSZKV6FNRAJQNLNQJTSH6AUXDDTLD2XCVBWXDKHYA, seg_pos={0 0}, upload=1236]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:104.36.82.54:4159): protocol: write tcp 159.223.143.39:55032->104.36.82.54:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:53.996583018 (1h21m9.777516394s) [noise=true , address=104.36.82.53:4133 , piece_id=FQ46TB5XSXDSDGDDVL7DJAN6UBGWM4FJ6IIKF76V2RWRECZZJTAQ, seg_pos={0 0}, upload=1236]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: write tcp 159.223.143.39:41768->104.36.82.53:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:53.996596896 (1h21m9.777530271s) [noise=true , address=104.36.82.47:4552 , piece_id=GU5YG4HBISSSZQVGDWXCIGKZU5J47RQXS2FIRI5R3JTTC2ZUNPDA, seg_pos={0 0}, upload=1236]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: write tcp 159.223.143.39:53628->104.36.82.47:4552: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:53.996627989 (1h21m9.777561373s) [seg_pos={0 0}, upload=1236]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:12:54.028909595 (1h21m9.809842963s) [upload=1236]: Done uploading segments... Jul 26 23:12:54.028937550 (1h21m9.809870916s) [upload=1236]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.034936367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X................X........................X......XXXXXXX] duration: 30.048262405s FAIL (2025-07-26 23:12:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.048284356s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X.X.X...XX.................X.X..........XX...X...X..XX.X...X] duration: 1m0.000824212s FAIL (2025-07-26 23:12:30 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000869975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X...................................................XXXXXXX] duration: 30.031930356s FAIL (2025-07-26 23:12:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 227]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:11:51.922205879 (1h20m7.703139258s) [upload=1227]: Done uploading segments... Jul 26 23:11:51.922237943 (1h20m7.703171324s) [upload=1227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:11:59.275064352 (1h20m15.055997735s) [upload=1238]: Starting upload Jul 26 23:11:59.275201024 (1h20m15.056134407s) [upload=1238]: Done starting upload Jul 26 23:11:59.275371411 (1h20m15.05630479s) [upload=1238]: Uploading segments... Jul 26 23:11:59.275820393 (1h20m15.056753769s) [upload=1238]: Got next segment. Inline: false Jul 26 23:11:59.275888085 (1h20m15.056821462s) [upload=1238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:11:59.275894320 (1h20m15.056827696s) [upload=1238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:11:59.369197868 (1h20m15.150131243s) [seg_pos={0 0}, upload=1238]: Begin upload segment... Jul 26 23:11:59.369251538 (1h20m15.150184919s) [seg_pos={0 0}, upload=1238]: Done begin upload segment. Jul 26 23:11:59.369313623 (1h20m15.150246999s) [noise=true , address=77.90.31.14:4661 , piece_id=L67XTU7HJEKUDA4VE73MYNYERV4ZALR4N7LJEYV2V6DJ63Q3K3CQ, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.369321582 (1h20m15.150254969s) [noise=true , address=45.91.116.4:4937 , piece_id=BQVYQ5YRGALYWP7BGYUBUMDRUIHDZBWCRYJPG6CSOBK63N6O6GQQ, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.369827640 (1h20m15.150761026s) [noise=true , address=104.36.82.6:4873 , piece_id=GNI5KXCPKVHSLFFTZOSUKBRMX4Q3FPHUMUKXWN7R4UBKHZIJTSJQ, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.370319128 (1h20m15.151252508s) [noise=true , address=45.91.116.5:4297 , piece_id=CXC3Y5NVJ7P3LITMVOYYC5XIA6T7NN7KL5Y6KIVUOGVWVKMUL6IA, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.370692150 (1h20m15.151625526s) [noise=true , address=104.36.82.8:4969 , piece_id=VU7QXL5CBVFM7PE2V5QFOUWEBXYX54DEBSAKQ577CDCCVPGANRRA, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.370980502 (1h20m15.15191388s) [noise=true , address=45.91.116.11:4119 , piece_id=JP7NC4YZMJRGMVMMOKTMXIZ4NOKCDZUZG44CNPY4U4QS4ND47WQQ, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.371348216 (1h20m15.152281604s) [noise=true , address=45.91.116.6:4973 , piece_id=QUO2Q7JA6M3KD6MM5GK6O7EABC25VMDY6424ANQ4T2CTMHLOBDEQ, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.371708287 (1h20m15.152641676s) [noise=true , address=77.90.31.18:4807 , piece_id=NZ6NAU6MXAK5AQYXYNBPU4QNR42JEFHMIXOJANCRPUSQY6BFBCBA, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.372233051 (1h20m15.153166435s) [noise=true , address=104.36.82.4:4803 , piece_id=TXNSO5PI4TA4PPYVJV5MACU4NOPFAL67XAUDMCM7EE3DKCWDKHNA, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.372645005 (1h20m15.153578388s) [noise=true , address=104.36.82.14:4222 , piece_id=WWQFSGCOA6ZSRYMS2X2EOJCC25PISRBPITYCKEZHP4LKC22NSJBQ, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.373035979 (1h20m15.153969373s) [noise=true , address=77.90.31.19:4842 , piece_id=YPABKMVA34UBARQ7FZGLBWC2HTIWQCAXL2IKRLELBNUHUU2ZQCUQ, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.373384991 (1h20m15.154318357s) [noise=true , address=45.91.116.8:4048 , piece_id=HFEF2LIR63WXGLQ2SJYBALOOJN5U2A2QBHWXNKMNTYAMYUICKKLA, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.373669159 (1h20m15.154602543s) [noise=true , address=104.36.82.2:4732 , piece_id=UX5YERCZJKJMYSYFX6MMSP5CESZAKUDO3VPMZSXLGDGYFJ6VT7TQ, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.373955118 (1h20m15.1548885s) [noise=true , address=104.36.82.6:4874 , piece_id=JU62TYWJDAQUOLOXGPL5JESPTYPVEEHTZJ6SVHAX32WAPH63AJCQ, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.374290659 (1h20m15.155224041s) [noise=true , address=104.36.82.9:4005 , piece_id=ATV6MIIOWLFWYOHAOSUTOBXCKRUB3OZ3KUOMKXLOAUYGPAGS6BOA, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.388647653 (1h20m15.169581033s) [noise=true , address=104.36.82.9:4007 , piece_id=AOLU2Q2AKMFVCOW4K7QBESE5E3FCX3VMW2FTIGP2JYXVOQWEBN7Q, seg_pos={0 0}, upload=1238]: Uploading piece... Jul 26 23:11:59.458545469 (1h20m15.239478849s) [upload=1238]: Next returned nil segment Jul 26 23:11:59.458565253 (1h20m15.239498638s) [upload=1238]: Waiting for error group managing segments... Jul 26 23:11:59.463050899 (1h20m15.243984264s) [noise=true , address=45.91.116.8:4048 , piece_id=HFEF2LIR63WXGLQ2SJYBALOOJN5U2A2QBHWXNKMNTYAMYUICKKLA, seg_pos={0 0}, upload=1238]: Done uploading piece. err: <nil> Jul 26 23:11:59.463601742 (1h20m15.244535127s) [noise=true , address=45.91.116.5:4297 , piece_id=CXC3Y5NVJ7P3LITMVOYYC5XIA6T7NN7KL5Y6KIVUOGVWVKMUL6IA, seg_pos={0 0}, upload=1238]: Done uploading piece. err: <nil> Jul 26 23:11:59.464187029 (1h20m15.245120394s) [noise=true , address=77.90.31.14:4661 , piece_id=L67XTU7HJEKUDA4VE73MYNYERV4ZALR4N7LJEYV2V6DJ63Q3K3CQ, seg_pos={0 0}, upload=1238]: Done uploading piece. err: <nil> Jul 26 23:11:59.464706701 (1h20m15.245640065s) [noise=true , address=45.91.116.11:4119 , piece_id=JP7NC4YZMJRGMVMMOKTMXIZ4NOKCDZUZG44CNPY4U4QS4ND47WQQ, seg_pos={0 0}, upload=1238]: Done uploading piece. err: <nil> Jul 26 23:11:59.465233576 (1h20m15.246166938s) [noise=true , address=45.91.116.6:4973 , piece_id=QUO2Q7JA6M3KD6MM5GK6O7EABC25VMDY6424ANQ4T2CTMHLOBDEQ, seg_pos={0 0}, upload=1238]: Done uploading piece. err: <nil> Jul 26 23:11:59.465873963 (1h20m15.246807329s) [noise=true , address=45.91.116.4:4937 , piece_id=BQVYQ5YRGALYWP7BGYUBUMDRUIHDZBWCRYJPG6CSOBK63N6O6GQQ, seg_pos={0 0}, upload=1238]: Done uploading piece. err: <nil> Jul 26 23:11:59.469923150 (1h20m15.250856519s) [noise=true , address=104.36.82.8:4969 , piece_id=VU7QXL5CBVFM7PE2V5QFOUWEBXYX54DEBSAKQ577CDCCVPGANRRA, seg_pos={0 0}, upload=1238]: Done uploading piece. err: <nil> Jul 26 23:11:59.514898702 (1h20m15.295832076s) [noise=true , address=77.90.31.18:4807 , piece_id=NZ6NAU6MXAK5AQYXYNBPU4QNR42JEFHMIXOJANCRPUSQY6BFBCBA, seg_pos={0 0}, upload=1238]: Done uploading piece. err: <nil> Jul 26 23:12:01.547481272 (1h20m17.328414659s) [noise=true , address=77.90.31.19:4842 , piece_id=YPABKMVA34UBARQ7FZGLBWC2HTIWQCAXL2IKRLELBNUHUU2ZQCUQ, seg_pos={0 0}, upload=1238]: Done uploading piece. err: <nil> Jul 26 23:12:29.226315048 (1h20m45.007248429s) [noise=true , address=104.36.82.2:4732 , piece_id=UX5YERCZJKJMYSYFX6MMSP5CESZAKUDO3VPMZSXLGDGYFJ6VT7TQ, seg_pos={0 0}, upload=1238]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:29.226444698 (1h20m45.007378076s) [noise=true , address=104.36.82.6:4874 , piece_id=JU62TYWJDAQUOLOXGPL5JESPTYPVEEHTZJ6SVHAX32WAPH63AJCQ, seg_pos={0 0}, upload=1238]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:29.226767495 (1h20m45.007700872s) [noise=true , address=104.36.82.6:4873 , piece_id=GNI5KXCPKVHSLFFTZOSUKBRMX4Q3FPHUMUKXWN7R4UBKHZIJTSJQ, seg_pos={0 0}, upload=1238]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:29.226832001 (1h20m45.00776538s) [noise=true , address=104.36.82.4:4803 , piece_id=TXNSO5PI4TA4PPYVJV5MACU4NOPFAL67XAUDMCM7EE3DKCWDKHNA, seg_pos={0 0}, upload=1238]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 159.223.143.39:40174->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:29.226896952 (1h20m45.00783033s) [noise=true , address=104.36.82.14:4222 , piece_id=WWQFSGCOA6ZSRYMS2X2EOJCC25PISRBPITYCKEZHP4LKC22NSJBQ, seg_pos={0 0}, upload=1238]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:33660->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:29.226974312 (1h20m45.00790769s) [noise=true , address=104.36.82.9:4005 , piece_id=ATV6MIIOWLFWYOHAOSUTOBXCKRUB3OZ3KUOMKXLOAUYGPAGS6BOA, seg_pos={0 0}, upload=1238]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:12:29.227128587 (1h20m45.008061954s) [noise=true , address=104.36.82.9:4007 , piece_id=AOLU2Q2AKMFVCOW4K7QBESE5E3FCX3VMW2FTIGP2JYXVOQWEBN7Q, seg_pos={0 0}, upload=1238]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:12:29.227149566 (1h20m45.008082932s) [seg_pos={0 0}, upload=1238]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:12:29.255589127 (1h20m45.036522523s) [upload=1238]: Done uploading segments... Jul 26 23:12:29.255618348 (1h20m45.036551714s) [upload=1238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.031965618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 269.961538ms FAIL (2025-07-26 23:12:22 UTC) orbiter warning: uplink: too many requests Duration: 269.999982ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XXX.X...] duration: 1.20062786s FAIL (2025-07-26 23:11:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.200660108s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXX.X..] duration: 13.160064691s FAIL (2025-07-26 23:11:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.16010322s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X...................................................XXXXXX] duration: 30.043447503s FAIL (2025-07-26 23:11:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== east 13 but got 6) Jul 26 23:11:21.914311815 (1h19m37.695245199s) [upload=1227]: Starting upload Jul 26 23:11:21.914407455 (1h19m37.69534083s) [upload=1227]: Done starting upload Jul 26 23:11:21.914524497 (1h19m37.695457872s) [upload=1227]: Uploading segments... Jul 26 23:11:21.914807350 (1h19m37.69574073s) [upload=1227]: Got next segment. Inline: false Jul 26 23:11:21.914877926 (1h19m37.695811309s) [upload=1227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:11:21.914884392 (1h19m37.69581779s) [upload=1227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:11:21.948402104 (1h19m37.729335482s) [seg_pos={0 0}, upload=1227]: Begin upload segment... Jul 26 23:11:21.948460644 (1h19m37.729394007s) [seg_pos={0 0}, upload=1227]: Done begin upload segment. Jul 26 23:11:21.948492232 (1h19m37.729425616s) [noise=true , address=45.91.116.8:4048 , piece_id=RQATIPIM5FWUQ5J5YQM5EQS3WCBFRDEYKXP64EA2NWQ4COOLBQVQ, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.948917868 (1h19m37.729851243s) [noise=true , address=45.91.116.14:4189 , piece_id=E2BQ4OU4HAKS4VBKXMXJXWELDUUEHMRFPWVCDXBVX44U2M5OFELQ, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.949226063 (1h19m37.730159442s) [noise=true , address=104.36.82.12:4151 , piece_id=U55C4RLDHQDNOWWBYY7MS4K6P2ACRJA2VN4MY5KMXAAUAMN46KDQ, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.949497866 (1h19m37.730431244s) [noise=true , address=104.36.82.4:4803 , piece_id=6PK7QZRCXEBCJ2R46ZUWHKDQHLE7U7U5VT4IVS5MGGYJAYWWLEIQ, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.950486970 (1h19m37.731420367s) [noise=true , address=45.91.116.5:4297 , piece_id=ZTRTK3HE6UAW5XSTMG5RXYO63H67H3Q2JKCYZZULLZEIMHZE5OXA, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.950905265 (1h19m37.731838652s) [noise=true , address=45.91.116.12:4226 , piece_id=YN3JHYTSRN4A6RJI44M5SKB4WSYSXIXOSOZ4MDV6JERKPUZKVJGQ, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.951320234 (1h19m37.732253612s) [noise=true , address=104.36.82.6:4873 , piece_id=6SIOZB7VL6LGOGQZNNRWSF5TKOWE3BMDRQT52T36W2IAXQ36U3LQ, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.951724437 (1h19m37.732657816s) [noise=true , address=45.91.116.13:4153 , piece_id=DXDFJEXR2MDFNIWLPSPAQJC6OXX3SUKYXYIRIEB2KHELDLCXIRIA, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.952131565 (1h19m37.733064946s) [noise=true , address=104.36.82.12:4150 , piece_id=HEYMD24YYX2WWPR6267N55W4MKUDHRQU6Q3FCK62Z2JB7NT24TCA, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.952547703 (1h19m37.733481082s) [noise=true , address=104.36.82.15:4257 , piece_id=QUL3GOOEGFIJPPCRFU3XPTLC2R4H4WIXRB4BAP23P2W7K5AGLDJA, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.952930222 (1h19m37.733863608s) [noise=true , address=104.36.82.9:4007 , piece_id=TKX5SUMPSMYWLLIPM5TWTJPOZLPJTVYMPPAXPAG5ZWCGNXKAZX2A, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.953353604 (1h19m37.734286991s) [noise=true , address=77.90.31.19:4842 , piece_id=HIWGNNN6IPJNYLZVEXKAHXKHYYRFQFRHRZTB7BK25FU5CZXHNVOA, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.954400839 (1h19m37.735334225s) [noise=true , address=45.91.116.14:4191 , piece_id=F53MASDT646KWUTKKAGM7KVGGRLB5FQQD23ABR5NBYPM6WRR4ZFQ, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.954924455 (1h19m37.73585784s) [noise=true , address=45.91.116.12:4227 , piece_id=CAUPKXGWV3WBAUE52XIAWGKCQAC55RXDOU5MPGW5HRDTLQFK5A6A, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.955475871 (1h19m37.736409236s) [noise=true , address=104.36.82.14:4222 , piece_id=MGGGQ3URI4RBT3Z3LN2HMNZHNK4N7ABBA2UEWZNMS6ONWUHRWLMA, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:21.955950676 (1h19m37.736884062s) [noise=true , address=104.36.82.8:4972 , piece_id=6H7FAQRMBAMXKR7ZKQVYZTFZNZYL4X2OZ2CM5XPESTOUMNNSQ4UQ, seg_pos={0 0}, upload=1227]: Uploading piece... Jul 26 23:11:22.041481822 (1h19m37.8224152s) [upload=1227]: Next returned nil segment Jul 26 23:11:22.041498512 (1h19m37.822431893s) [upload=1227]: Waiting for error group managing segments... Jul 26 23:11:22.045203343 (1h19m37.82613671s) [noise=true , address=45.91.116.8:4048 , piece_id=RQATIPIM5FWUQ5J5YQM5EQS3WCBFRDEYKXP64EA2NWQ4COOLBQVQ, seg_pos={0 0}, upload=1227]: Done uploading piece. err: <nil> Jul 26 23:11:22.045247655 (1h19m37.826181026s) [noise=true , address=45.91.116.12:4227 , piece_id=CAUPKXGWV3WBAUE52XIAWGKCQAC55RXDOU5MPGW5HRDTLQFK5A6A, seg_pos={0 0}, upload=1227]: Done uploading piece. err: <nil> Jul 26 23:11:22.045784652 (1h19m37.826718019s) [noise=true , address=45.91.116.14:4191 , piece_id=F53MASDT646KWUTKKAGM7KVGGRLB5FQQD23ABR5NBYPM6WRR4ZFQ, seg_pos={0 0}, upload=1227]: Done uploading piece. err: <nil> Jul 26 23:11:22.045824317 (1h19m37.826757679s) [noise=true , address=45.91.116.12:4226 , piece_id=YN3JHYTSRN4A6RJI44M5SKB4WSYSXIXOSOZ4MDV6JERKPUZKVJGQ, seg_pos={0 0}, upload=1227]: Done uploading piece. err: <nil> Jul 26 23:11:22.046216443 (1h19m37.827149813s) [noise=true , address=45.91.116.5:4297 , piece_id=ZTRTK3HE6UAW5XSTMG5RXYO63H67H3Q2JKCYZZULLZEIMHZE5OXA, seg_pos={0 0}, upload=1227]: Done uploading piece. err: <nil> Jul 26 23:11:22.046395648 (1h19m37.827329028s) [noise=true , address=45.91.116.13:4153 , piece_id=DXDFJEXR2MDFNIWLPSPAQJC6OXX3SUKYXYIRIEB2KHELDLCXIRIA, seg_pos={0 0}, upload=1227]: Done uploading piece. err: <nil> Jul 26 23:11:22.046612412 (1h19m37.827545778s) [noise=true , address=45.91.116.14:4189 , piece_id=E2BQ4OU4HAKS4VBKXMXJXWELDUUEHMRFPWVCDXBVX44U2M5OFELQ, seg_pos={0 0}, upload=1227]: Done uploading piece. err: <nil> Jul 26 23:11:23.825157758 (1h19m39.606091119s) [noise=true , address=77.90.31.19:4842 , piece_id=HIWGNNN6IPJNYLZVEXKAHXKHYYRFQFRHRZTB7BK25FU5CZXHNVOA, seg_pos={0 0}, upload=1227]: Done uploading piece. err: <nil> Jul 26 23:11:51.881720635 (1h20m7.662654016s) [noise=true , address=104.36.82.12:4150 , piece_id=HEYMD24YYX2WWPR6267N55W4MKUDHRQU6Q3FCK62Z2JB7NT24TCA, seg_pos={0 0}, upload=1227]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:51.881815758 (1h20m7.662749127s) [noise=true , address=104.36.82.9:4007 , piece_id=TKX5SUMPSMYWLLIPM5TWTJPOZLPJTVYMPPAXPAG5ZWCGNXKAZX2A, seg_pos={0 0}, upload=1227]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:37536->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:51.881889146 (1h20m7.662822512s) [noise=true , address=104.36.82.4:4803 , piece_id=6PK7QZRCXEBCJ2R46ZUWHKDQHLE7U7U5VT4IVS5MGGYJAYWWLEIQ, seg_pos={0 0}, upload=1227]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 159.223.143.39:55794->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:51.881958995 (1h20m7.662892373s) [noise=true , address=104.36.82.6:4873 , piece_id=6SIOZB7VL6LGOGQZNNRWSF5TKOWE3BMDRQT52T36W2IAXQ36U3LQ, seg_pos={0 0}, upload=1227]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:51520->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:51.882041587 (1h20m7.662974956s) [noise=true , address=104.36.82.15:4257 , piece_id=QUL3GOOEGFIJPPCRFU3XPTLC2R4H4WIXRB4BAP23P2W7K5AGLDJA, seg_pos={0 0}, upload=1227]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:51796->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:51.882087324 (1h20m7.663020692s) [noise=true , address=104.36.82.14:4222 , piece_id=MGGGQ3URI4RBT3Z3LN2HMNZHNK4N7ABBA2UEWZNMS6ONWUHRWLMA, seg_pos={0 0}, upload=1227]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:51.882120908 (1h20m7.663054276s) [noise=true , address=104.36.82.8:4972 , piece_id=6H7FAQRMBAMXKR7ZKQVYZTFZNZYL4X2OZ2CM5XPESTOUMNNSQ4UQ, seg_pos={0 0}, upload=1227]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:51.882163652 (1h20m7.663097019s) [noise=true , address=104.36.82.12:4151 , piece_id=U55C4RLDHQDNOWWBYY7MS4K6P2ACRJA2VN4MY5KMXAAUAMN46KDQ, seg_pos={0 0}, upload=1227]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:51.882184715 (1h20m7.663118098s) [seg_pos={0 0}, upload=1227]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:11:51.922205879 (1h20m7.703139258s) [upload=1227]: Done uploading segments... Jul 26 23:11:51.922237943 (1h20m7.703171324s) [upload=1227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.043477612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X................X........................X......XXXXXX] duration: 30.032981169s FAIL (2025-07-26 23:11:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.033008481s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 288.37581ms FAIL (2025-07-26 23:11:42 UTC) orbiter warning: uplink: too many requests Duration: 288.416516ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)