Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X.........................X.........X...........] duration: 10.99412872s FAIL (2025-06-22 00:22:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.994148205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.X................] duration: 21.97704943s FAIL (2025-06-22 00:21:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.977081413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X.........................X.........X..........] duration: 18.342960694s FAIL (2025-06-22 00:21:26 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.342984911s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X.........................X.........X.........] duration: 23.574565056s FAIL (2025-06-22 00:21:08 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.57459659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.X...............] duration: 20.269304223s FAIL (2025-06-22 00:21:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.269339422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.X..............] duration: 21.276192875s FAIL (2025-06-22 00:20:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.27621753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 41.30152168s FAIL (2025-06-22 00:20:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 41.301568181s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X.........................X.........X........] duration: 13.557390625s FAIL (2025-06-22 00:20:25 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.557466512s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X.........................X.........X.......] duration: 46.537234535s FAIL (2025-06-22 00:19:56 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.537284338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.X............] duration: 34.596080624s FAIL (2025-06-22 00:19:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.59610508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 48.935574617s FAIL (2025-06-22 00:19:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 48.935605173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X.........................X.........X......] duration: 11.265711418s FAIL (2025-06-22 00:18:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.265728201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X.........................X.........X.....] duration: 49.142722129s FAIL (2025-06-22 00:18:11 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.143443241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 17.091964786s FAIL (2025-06-22 00:17:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.092002059s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................X.........................X.........X....] duration: 24.423376486s FAIL (2025-06-22 00:17:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.423413757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 26.048462453s FAIL (2025-06-22 00:16:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.048553108s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................X.........................X.........X...] duration: 15.256340559s FAIL (2025-06-22 00:15:58 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.256361849s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 15.238646549s FAIL (2025-06-22 00:15:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.238677023s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.........................X.........X..] duration: 18.157164233s FAIL (2025-06-22 00:15:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.157186793s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................X.........................X.........X.] duration: 41.520421441s FAIL (2025-06-22 00:14:59 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.52045217s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 24.840845305s FAIL (2025-06-22 00:14:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.840897073s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.........................X.........X] duration: 1m0.001608906s FAIL (2025-06-22 00:14:16 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001635636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 35.476488457s FAIL (2025-06-22 00:14:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 35.476509556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 35.390691655s FAIL (2025-06-22 00:13:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 35.390714086s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X.........................X.........] duration: 23.317726532s FAIL (2025-06-22 00:13:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.31776193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.........................X........] duration: 19.478306111s FAIL (2025-06-22 00:12:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.478324637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 28.057237775s FAIL (2025-06-22 00:12:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.057288269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.........................X.......] duration: 19.201999681s FAIL (2025-06-22 00:11:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.202027837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 36.378979824s FAIL (2025-06-22 00:11:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 36.379055155s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.........................X......] duration: 22.261023884s FAIL (2025-06-22 00:11:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.261071658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 51.383479816s FAIL (2025-06-22 00:10:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 51.383508606s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X.........................X.....] duration: 15.546862434s FAIL (2025-06-22 00:10:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.546899019s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.........................X....] duration: 41.689439936s FAIL (2025-06-22 00:10:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.689455717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 23.378014006s FAIL (2025-06-22 00:09:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.378037387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.........................X...] duration: 35.19722223s FAIL (2025-06-22 00:09:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.197250316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 1m0.024279568s FAIL (2025-06-22 00:09:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== g_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324082782 (79h44m51.88740579s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324387615 (79h44m51.887710618s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324693763 (79h44m51.888016771s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.324987498 (79h44m51.888310498s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.325315636 (79h44m51.888638643s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:08:16.394058454 (79h44m51.957381445s) [upload=63843]: Next returned nil segment Jun 22 00:08:16.394072985 (79h44m51.957395986s) [upload=63843]: Waiting for error group managing segments... Jun 22 00:08:16.397574195 (79h44m51.960897187s) [noise=true , address=45.91.116.5:5308 , piece_id=QZA34FLZEJNVE2TUHVF5ZMTG2WP7RTNNOJF7AZ4YLYCA22XJPCZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:16.459349430 (79h44m52.022672432s) [noise=true , address=104.36.82.54:4173 , piece_id=O2FBSZLX7FIDIDJWGWV2RMIZ7DZ5WYKS7JLZIOK64D5TRKSP3U7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:16.518220521 (79h44m52.081543528s) [noise=true , address=77.90.31.12:4605 , piece_id=AITH3LEQCYAWVX4JXU7LDBV4LRIA3Y2LI3FNTOISOMRQ4LWNZ2VQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:17.014258958 (79h44m52.577581952s) [noise=true , address=64.49.13.137:4500 , piece_id=FMD3ECZINI2YWY32VYSH7GKCITBALAEDUMYCNTWTAJQVY55Y2MLQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:15TwVQ6CHDbtqS4XW86n9BuyjdqpaJo4ZKPgPnmhSm6EPDrnd9, address:64.49.13.137:4500): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.153129516 (79h44m52.716452522s) [noise=true , address=104.36.82.16:6328 , piece_id=4TEPB6TBOWSY5KR52EVGRYVGHAKBEDTUHMFF3KJPQKZXYRGDRBJQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:6328): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.275208950 (79h44m52.83853195s) [noise=true , address=79.127.205.235:5664 , piece_id=XB53MDJKRRNXK6SAMHX2RHAEO6SAHTKI7732ERADDN7JCWDZKNVA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.605904252 (79h44m53.169227253s) [noise=true , address=77.90.31.21:4887 , piece_id=3JB4YN5JQ2KHD3LXGYAH2VQDBYAM63SYBNIVNUK2BAVW6TG6L6EA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:08:17.865919302 (79h44m53.429242294s) [noise=true , address=77.90.31.14:4693 , piece_id=QAVXJYMYM3XOYRGUPHMK257ZNL7CUMR6UBGSJM647C73ECJDCVIA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:19.517357026 (79h44m55.080680046s) [noise=true , address=104.36.82.10:6108 , piece_id=QTWYQJQ22NWG7I6WRKMC6AL7YYPHVEBN2LIUVIFZHC5WMDPKPANQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.580254458 (79h44m56.143577448s) [noise=true , address=104.36.82.7:5938 , piece_id=YRRM6JLCSPEV2UZI5CICTMVPW2MTIK7P5K5F5S53AEV2DEMJFA3Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:20.752577874 (79h44m56.315900867s) [noise=true , address=104.36.82.20:6466 , piece_id=P7UZBSRQEY3ORJAXMYSNYBPTYNGDBSSKZL74Z2GSJNMON3VZBOZQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:24.515800663 (79h45m0.079123665s) [noise=true , address=79.127.219.38:4028 , piece_id=QZDOXSRJI6MXMIUNJLVRKZBSCRMDI54KPFDSR4D4JF6S2RZ4SV7A, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:25.948556776 (79h45m1.511879777s) [noise=true , address=104.36.82.21:6473 , piece_id=3M34BSPUAFRF5NXPWIXKH25KX67RCHVXQCS5MFS5S7X7RYF676RQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:32.636880825 (79h45m8.200203818s) [noise=true , address=104.36.82.56:4261 , piece_id=2RGJ5T5RFUR67IIUA5CUVXFL4RVBO7GVLDLAVUZCRHWAITSBOEWA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:35.525047894 (79h45m11.088370884s) [noise=true , address=104.36.82.19:6406 , piece_id=ERRJIBJ6LCOK3QRXCJSASUGTFUL76SS3HZJI2VQ64HWTSUMGVHZA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:40.407828721 (79h45m15.971151709s) [noise=true , address=104.36.82.14:6234 , piece_id=WUFOSCTTAKQZLII6GTESOCKX4MJZM6CA45XBQUUKGS6S7JQASLUQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:08:52.755279221 (79h45m28.318602228s) [noise=true , address=104.36.82.52:4103 , piece_id=MWVPOZMAAFPZD6HDK4SGJOKW5IMNYG75R3YA7LOLMNJVQOLP3SBQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.049083116 (79h45m51.612406117s) [noise=true , address=104.36.82.55:4219 , piece_id=BMAN2MOGHY3DLY5O3GYQUSAUQ7A77YSTD7GEHUAUSSPE5EE7LJQQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: <nil> Jun 22 00:09:16.067949159 (79h45m51.631272164s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.067962327 (79h45m51.63128533s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068067279 (79h45m51.631390289s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068445650 (79h45m51.631768659s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.068751557 (79h45m51.632074575s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Uploading piece... Jun 22 00:09:16.254420059 (79h45m51.817743058s) [noise=true , address=45.91.116.7:5012 , piece_id=7TSTZEQJHXFCT4UIUEOF6Q3JML3YJRJRJWHO3ZYPGVY66BCSDZ7Q, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:5012): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 00:09:16.254721861 (79h45m51.818044861s) [noise=true , address=104.36.82.11:6139 , piece_id=QQUWKI2WTCYU44QDNA4OOBWSPVTCMMMLEXEPOASV35A4UZVPW3TQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:6139): protocol: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45694->104.36.82.11:6139: use of closed network connection Jun 22 00:09:16.254797295 (79h45m51.818120295s) [noise=true , address=104.36.82.20:6468 , piece_id=SQUT67FBQGV423UW4LOAECRWUJWCJ7AQ4ED3ACD3MZUTBRXUSYMA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:123Cc2RmkYzkkbder6auanNvu1sTSEM6QD5aagnNC28DviG41Gi, address:104.36.82.20:6468): protocol: read tcp 159.223.143.39:35576->104.36.82.20:6468: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254808564 (79h45m51.818131557s) [noise=true , address=77.90.31.17:4803 , piece_id=DXJWERKQZP445OUZQF6W33T5JXREOIMFYAMWDMDDMJS736PXWAXQ, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: read tcp 159.223.143.39:42574->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 22 00:09:16.254859521 (79h45m51.81818252s) [noise=true , address=79.127.205.235:5664 , piece_id=NXG2ACQ4GJGVX7PQYTR5MCRWRPK6Y5FXNUN7YVETYC3U4ROXMOFA, seg_pos={0 0}, upload=63843]: Done uploading piece. err: ecclient: upload failed (node:12p8Gg6yYNUbFiEkHLHDFdnPC2w4AHgguBYv2ce34AYWXQzhHER, address:79.127.205.235:5664): protocol: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54186->79.127.205.235:5664: use of closed network connection Jun 22 00:09:16.254895863 (79h45m51.818218866s) [seg_pos={0 0}, upload=63843]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:09:16.276852311 (79h45m51.840175305s) [upload=63843]: Done uploading segments... Jun 22 00:09:16.276903113 (79h45m51.840226102s) [upload=63843]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.024303991s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.........................X..] duration: 15.0738305s FAIL (2025-06-22 00:08:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.073869979s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X.........................X.] duration: 24.411107392s FAIL (2025-06-22 00:08:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.411137422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 18.914771787s FAIL (2025-06-22 00:07:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 05:59.086456248 (79h42m34.649779236s) [upload=63806]: Upload finished. err: <nil> Jun 22 00:06:35.070905470 (79h43m10.634228474s) [upload=63815]: Starting upload Jun 22 00:06:35.071010688 (79h43m10.63433369s) [upload=63815]: Done starting upload Jun 22 00:06:35.071201792 (79h43m10.634524782s) [upload=63815]: Uploading segments... Jun 22 00:06:35.071643829 (79h43m10.634966841s) [upload=63815]: Got next segment. Inline: false Jun 22 00:06:35.071764824 (79h43m10.63508783s) [upload=63815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:06:35.071774624 (79h43m10.635097634s) [upload=63815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:06:35.107204892 (79h43m10.670527901s) [seg_pos={0 0}, upload=63815]: Begin upload segment... Jun 22 00:06:35.107425462 (79h43m10.670748472s) [noise=true , address=104.36.82.42:6803 , piece_id=DZ5E65IJ34B66MYV7MYQ4TIJE44PWIWAKGYRCVDPYGTLKF6UKN4A, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.107974355 (79h43m10.671297357s) [noise=true , address=77.90.31.14:4692 , piece_id=O3GBKLZWYWSUKTZONTAFH3GPZFGTQTRQELSCMT6KB6LZFKETGKVQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.108920911 (79h43m10.672243923s) [noise=true , address=104.36.82.21:6493 , piece_id=GGBVGKVSAWFYTLYPNME4EBY3QAMQFA2MESSDNWAG7WLAD3NCPHKA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.109421565 (79h43m10.672744573s) [noise=true , address=104.36.82.20:6468 , piece_id=CCSQEOUXRJC7AXDSH4EASAPPSK7365ZA72OZI56C4NTVL44U46WQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.109844294 (79h43m10.673167296s) [noise=true , address=79.127.205.234:5611 , piece_id=6NNVTKVHHZVN6BI34CVNW5DPLE7HI4F52B3TMASMKWKRM43D4QVQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.110267970 (79h43m10.673590974s) [noise=true , address=79.127.219.38:4036 , piece_id=SSRGW7SWOVNO4XSRIKLNDXW5F2R46FTOFQJNUAQQ34PILKHX3I4A, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.111464573 (79h43m10.674787575s) [seg_pos={0 0}, upload=63815]: Done begin upload segment. Jun 22 00:06:35.111520150 (79h43m10.674843159s) [noise=true , address=77.90.31.17:4803 , piece_id=MA4QUVSB5MBKWVHZSU2HTSD7HNKFCDM5LH6AOMW2JG2VJAUPNWJQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.112035917 (79h43m10.675358922s) [noise=true , address=79.127.220.98:4402 , piece_id=TYEZTCGQRZTBHMZZDDZQTNG4DZCTFIJ3REAFDSOQWZW3UHWKSJTA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.112494883 (79h43m10.675817888s) [noise=true , address=104.36.82.55:4204 , piece_id=TII6GSL5AQ65NVLWSTR4FV4Y3KIGOZTH6VKYSFVK3SNAQ4VAM4HA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.112954016 (79h43m10.676277025s) [noise=true , address=104.36.82.13:6196 , piece_id=IGOXQ3NJ5IDXUMUDQS7LVSSFVLZHHDL4ZDUQRCRT3SGDP6C2WBFA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.113391990 (79h43m10.676714991s) [noise=true , address=104.36.82.53:4146 , piece_id=3R7U2WZVOFQK3YAMMBQFB62JEV4GO72YSSKZHGWUCOLGSHPYWVEQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.113834067 (79h43m10.677157075s) [noise=true , address=104.36.82.5:5848 , piece_id=RESM36SSKLRSRBTHF24KRJYXRLVJ2EGYLZYMOMAERDSBHQNGZHTQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.114273706 (79h43m10.677596718s) [noise=true , address=104.36.82.52:4108 , piece_id=CK6YCPRZFTVJNOBD6WJGXIFXJ3P5ZTKGK76CTN4HGJ3GWVYM6XPQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.114737587 (79h43m10.678060595s) [noise=true , address=77.90.31.21:4887 , piece_id=OOVHZJWHOBHJUBWPZBJ7JFHAVCONGWOIQKNSWRUDRLMUVDJVW3JA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.123351328 (79h43m10.68667432s) [noise=true , address=104.36.82.54:4186 , piece_id=UWTOIEDDR76OM6Y2ACCRIUTAPMMMR7MDZZJ4WKWR5DJ55IQQ67QA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.124034043 (79h43m10.68735705s) [noise=true , address=104.36.82.56:4241 , piece_id=YJESR3E5OCIDCHNDKB4EO5MPKH5IP4WIWP5FBLVF7A4SSWIDMPDQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.124489284 (79h43m10.687812288s) [noise=true , address=77.90.31.18:4808 , piece_id=FIJJMZP53YXE7OJ33B5EAVPPA2G2QRMGPO3LS7VCZO55UJIFFYSQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.124937699 (79h43m10.688260705s) [noise=true , address=156.146.43.225:4459 , piece_id=WBTUFC6GX6I7ABGQKIXP7VXPHIDI5TLDORBT2AFVFGV635JTCA4A, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.219375282 (79h43m10.782698289s) [upload=63815]: Next returned nil segment Jun 22 00:06:35.219398254 (79h43m10.782721278s) [upload=63815]: Waiting for error group managing segments... Jun 22 00:06:35.228097103 (79h43m10.791420115s) [noise=true , address=79.127.220.98:4402 , piece_id=TYEZTCGQRZTBHMZZDDZQTNG4DZCTFIJ3REAFDSOQWZW3UHWKSJTA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: ecclient: upload failed (node:1ko3uf7rucXvxwRA1mew5Tyyn9r1wzwffnay9Anicvq48fCWoP, address:79.127.220.98:4402): protocol: storage node overloaded, request limit: 500 Jun 22 00:06:35.273790699 (79h43m10.837113702s) [noise=true , address=77.90.31.18:4808 , piece_id=FIJJMZP53YXE7OJ33B5EAVPPA2G2QRMGPO3LS7VCZO55UJIFFYSQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:35.496191663 (79h43m11.05951466s) [noise=true , address=156.146.43.225:4459 , piece_id=WBTUFC6GX6I7ABGQKIXP7VXPHIDI5TLDORBT2AFVFGV635JTCA4A, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:35.508506927 (79h43m11.07182992s) [noise=true , address=79.127.205.234:5611 , piece_id=6NNVTKVHHZVN6BI34CVNW5DPLE7HI4F52B3TMASMKWKRM43D4QVQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: ecclient: upload failed (node:1Zck81G1h1gLrVPahfQ7VhtPdM3ciR2MsSkmttjYgaFuZDKvT5, address:79.127.205.234:5611): protocol: storage node overloaded, request limit: 2000 Jun 22 00:06:35.775728264 (79h43m11.339051266s) [noise=true , address=77.90.31.17:4803 , piece_id=MA4QUVSB5MBKWVHZSU2HTSD7HNKFCDM5LH6AOMW2JG2VJAUPNWJQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: storage node overloaded, request limit: 2000 Jun 22 00:06:35.915798691 (79h43m11.479121694s) [noise=true , address=104.36.82.42:6803 , piece_id=DZ5E65IJ34B66MYV7MYQ4TIJE44PWIWAKGYRCVDPYGTLKF6UKN4A, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:35.981958943 (79h43m11.545281937s) [noise=true , address=77.90.31.14:4692 , piece_id=O3GBKLZWYWSUKTZONTAFH3GPZFGTQTRQELSCMT6KB6LZFKETGKVQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:36.542263970 (79h43m12.105586963s) [noise=true , address=104.36.82.13:6196 , piece_id=IGOXQ3NJ5IDXUMUDQS7LVSSFVLZHHDL4ZDUQRCRT3SGDP6C2WBFA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:37.106276338 (79h43m12.669599339s) [noise=true , address=77.90.31.21:4887 , piece_id=OOVHZJWHOBHJUBWPZBJ7JFHAVCONGWOIQKNSWRUDRLMUVDJVW3JA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:06:38.655697328 (79h43m14.219020328s) [noise=true , address=104.36.82.5:5848 , piece_id=RESM36SSKLRSRBTHF24KRJYXRLVJ2EGYLZYMOMAERDSBHQNGZHTQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:42.966609148 (79h43m18.529932132s) [noise=true , address=104.36.82.20:6468 , piece_id=CCSQEOUXRJC7AXDSH4EASAPPSK7365ZA72OZI56C4NTVL44U46WQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:43.732966458 (79h43m19.296289462s) [noise=true , address=79.127.219.38:4036 , piece_id=SSRGW7SWOVNO4XSRIKLNDXW5F2R46FTOFQJNUAQQ34PILKHX3I4A, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:45.867417843 (79h43m21.430740847s) [noise=true , address=104.36.82.21:6493 , piece_id=GGBVGKVSAWFYTLYPNME4EBY3QAMQFA2MESSDNWAG7WLAD3NCPHKA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:50.615388344 (79h43m26.178711347s) [noise=true , address=104.36.82.53:4146 , piece_id=3R7U2WZVOFQK3YAMMBQFB62JEV4GO72YSSKZHGWUCOLGSHPYWVEQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:51.372429136 (79h43m26.935752124s) [noise=true , address=104.36.82.56:4241 , piece_id=YJESR3E5OCIDCHNDKB4EO5MPKH5IP4WIWP5FBLVF7A4SSWIDMPDQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:54.589773515 (79h43m30.153096515s) [noise=true , address=104.36.82.54:4186 , piece_id=UWTOIEDDR76OM6Y2ACCRIUTAPMMMR7MDZZJ4WKWR5DJ55IQQ67QA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:56.923953508 (79h43m32.487276509s) [noise=true , address=104.36.82.55:4204 , piece_id=TII6GSL5AQ65NVLWSTR4FV4Y3KIGOZTH6VKYSFVK3SNAQ4VAM4HA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:07:35.039734848 (79h44m10.603057859s) [noise=true , address=104.36.82.52:4108 , piece_id=CK6YCPRZFTVJNOBD6WJGXIFXJ3P5ZTKGK76CTN4HGJ3GWVYM6XPQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: ecclient: upload failed (node:12XJJZCUSHUS52YC6jhoPB4ayjsyKZT477PymPj4inkZtdUvheT, address:104.36.82.52:4108): protocol: write tcp 159.223.143.39:34012->104.36.82.52:4108: use of closed network connection; write tcp 159.223.143.39:34012->104.36.82.52:4108: use of closed network connection Jun 22 00:07:35.039830671 (79h44m10.603153681s) [seg_pos={0 0}, upload=63815]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:07:35.068529987 (79h44m10.631852995s) [upload=63815]: Done uploading segments... Jun 22 00:07:35.068576498 (79h44m10.631899502s) [upload=63815]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.914796574s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.........................X] duration: 1m0.001383204s FAIL (2025-06-22 00:07:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001405034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.031383366s FAIL (2025-06-22 00:07:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 05:59.086456248 (79h42m34.649779236s) [upload=63806]: Upload finished. err: <nil> Jun 22 00:06:35.070905470 (79h43m10.634228474s) [upload=63815]: Starting upload Jun 22 00:06:35.071010688 (79h43m10.63433369s) [upload=63815]: Done starting upload Jun 22 00:06:35.071201792 (79h43m10.634524782s) [upload=63815]: Uploading segments... Jun 22 00:06:35.071643829 (79h43m10.634966841s) [upload=63815]: Got next segment. Inline: false Jun 22 00:06:35.071764824 (79h43m10.63508783s) [upload=63815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 00:06:35.071774624 (79h43m10.635097634s) [upload=63815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 00:06:35.107204892 (79h43m10.670527901s) [seg_pos={0 0}, upload=63815]: Begin upload segment... Jun 22 00:06:35.107425462 (79h43m10.670748472s) [noise=true , address=104.36.82.42:6803 , piece_id=DZ5E65IJ34B66MYV7MYQ4TIJE44PWIWAKGYRCVDPYGTLKF6UKN4A, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.107974355 (79h43m10.671297357s) [noise=true , address=77.90.31.14:4692 , piece_id=O3GBKLZWYWSUKTZONTAFH3GPZFGTQTRQELSCMT6KB6LZFKETGKVQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.108920911 (79h43m10.672243923s) [noise=true , address=104.36.82.21:6493 , piece_id=GGBVGKVSAWFYTLYPNME4EBY3QAMQFA2MESSDNWAG7WLAD3NCPHKA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.109421565 (79h43m10.672744573s) [noise=true , address=104.36.82.20:6468 , piece_id=CCSQEOUXRJC7AXDSH4EASAPPSK7365ZA72OZI56C4NTVL44U46WQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.109844294 (79h43m10.673167296s) [noise=true , address=79.127.205.234:5611 , piece_id=6NNVTKVHHZVN6BI34CVNW5DPLE7HI4F52B3TMASMKWKRM43D4QVQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.110267970 (79h43m10.673590974s) [noise=true , address=79.127.219.38:4036 , piece_id=SSRGW7SWOVNO4XSRIKLNDXW5F2R46FTOFQJNUAQQ34PILKHX3I4A, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.111464573 (79h43m10.674787575s) [seg_pos={0 0}, upload=63815]: Done begin upload segment. Jun 22 00:06:35.111520150 (79h43m10.674843159s) [noise=true , address=77.90.31.17:4803 , piece_id=MA4QUVSB5MBKWVHZSU2HTSD7HNKFCDM5LH6AOMW2JG2VJAUPNWJQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.112035917 (79h43m10.675358922s) [noise=true , address=79.127.220.98:4402 , piece_id=TYEZTCGQRZTBHMZZDDZQTNG4DZCTFIJ3REAFDSOQWZW3UHWKSJTA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.112494883 (79h43m10.675817888s) [noise=true , address=104.36.82.55:4204 , piece_id=TII6GSL5AQ65NVLWSTR4FV4Y3KIGOZTH6VKYSFVK3SNAQ4VAM4HA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.112954016 (79h43m10.676277025s) [noise=true , address=104.36.82.13:6196 , piece_id=IGOXQ3NJ5IDXUMUDQS7LVSSFVLZHHDL4ZDUQRCRT3SGDP6C2WBFA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.113391990 (79h43m10.676714991s) [noise=true , address=104.36.82.53:4146 , piece_id=3R7U2WZVOFQK3YAMMBQFB62JEV4GO72YSSKZHGWUCOLGSHPYWVEQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.113834067 (79h43m10.677157075s) [noise=true , address=104.36.82.5:5848 , piece_id=RESM36SSKLRSRBTHF24KRJYXRLVJ2EGYLZYMOMAERDSBHQNGZHTQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.114273706 (79h43m10.677596718s) [noise=true , address=104.36.82.52:4108 , piece_id=CK6YCPRZFTVJNOBD6WJGXIFXJ3P5ZTKGK76CTN4HGJ3GWVYM6XPQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.114737587 (79h43m10.678060595s) [noise=true , address=77.90.31.21:4887 , piece_id=OOVHZJWHOBHJUBWPZBJ7JFHAVCONGWOIQKNSWRUDRLMUVDJVW3JA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.123351328 (79h43m10.68667432s) [noise=true , address=104.36.82.54:4186 , piece_id=UWTOIEDDR76OM6Y2ACCRIUTAPMMMR7MDZZJ4WKWR5DJ55IQQ67QA, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.124034043 (79h43m10.68735705s) [noise=true , address=104.36.82.56:4241 , piece_id=YJESR3E5OCIDCHNDKB4EO5MPKH5IP4WIWP5FBLVF7A4SSWIDMPDQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.124489284 (79h43m10.687812288s) [noise=true , address=77.90.31.18:4808 , piece_id=FIJJMZP53YXE7OJ33B5EAVPPA2G2QRMGPO3LS7VCZO55UJIFFYSQ, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.124937699 (79h43m10.688260705s) [noise=true , address=156.146.43.225:4459 , piece_id=WBTUFC6GX6I7ABGQKIXP7VXPHIDI5TLDORBT2AFVFGV635JTCA4A, seg_pos={0 0}, upload=63815]: Uploading piece... Jun 22 00:06:35.219375282 (79h43m10.782698289s) [upload=63815]: Next returned nil segment Jun 22 00:06:35.219398254 (79h43m10.782721278s) [upload=63815]: Waiting for error group managing segments... Jun 22 00:06:35.228097103 (79h43m10.791420115s) [noise=true , address=79.127.220.98:4402 , piece_id=TYEZTCGQRZTBHMZZDDZQTNG4DZCTFIJ3REAFDSOQWZW3UHWKSJTA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: ecclient: upload failed (node:1ko3uf7rucXvxwRA1mew5Tyyn9r1wzwffnay9Anicvq48fCWoP, address:79.127.220.98:4402): protocol: storage node overloaded, request limit: 500 Jun 22 00:06:35.273790699 (79h43m10.837113702s) [noise=true , address=77.90.31.18:4808 , piece_id=FIJJMZP53YXE7OJ33B5EAVPPA2G2QRMGPO3LS7VCZO55UJIFFYSQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:35.496191663 (79h43m11.05951466s) [noise=true , address=156.146.43.225:4459 , piece_id=WBTUFC6GX6I7ABGQKIXP7VXPHIDI5TLDORBT2AFVFGV635JTCA4A, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:35.508506927 (79h43m11.07182992s) [noise=true , address=79.127.205.234:5611 , piece_id=6NNVTKVHHZVN6BI34CVNW5DPLE7HI4F52B3TMASMKWKRM43D4QVQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: ecclient: upload failed (node:1Zck81G1h1gLrVPahfQ7VhtPdM3ciR2MsSkmttjYgaFuZDKvT5, address:79.127.205.234:5611): protocol: storage node overloaded, request limit: 2000 Jun 22 00:06:35.775728264 (79h43m11.339051266s) [noise=true , address=77.90.31.17:4803 , piece_id=MA4QUVSB5MBKWVHZSU2HTSD7HNKFCDM5LH6AOMW2JG2VJAUPNWJQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: storage node overloaded, request limit: 2000 Jun 22 00:06:35.915798691 (79h43m11.479121694s) [noise=true , address=104.36.82.42:6803 , piece_id=DZ5E65IJ34B66MYV7MYQ4TIJE44PWIWAKGYRCVDPYGTLKF6UKN4A, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:35.981958943 (79h43m11.545281937s) [noise=true , address=77.90.31.14:4692 , piece_id=O3GBKLZWYWSUKTZONTAFH3GPZFGTQTRQELSCMT6KB6LZFKETGKVQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:36.542263970 (79h43m12.105586963s) [noise=true , address=104.36.82.13:6196 , piece_id=IGOXQ3NJ5IDXUMUDQS7LVSSFVLZHHDL4ZDUQRCRT3SGDP6C2WBFA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:37.106276338 (79h43m12.669599339s) [noise=true , address=77.90.31.21:4887 , piece_id=OOVHZJWHOBHJUBWPZBJ7JFHAVCONGWOIQKNSWRUDRLMUVDJVW3JA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: ecclient: upload failed (node:12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, address:77.90.31.21:4887): protocol: storage node overloaded, request limit: 2000 Jun 22 00:06:38.655697328 (79h43m14.219020328s) [noise=true , address=104.36.82.5:5848 , piece_id=RESM36SSKLRSRBTHF24KRJYXRLVJ2EGYLZYMOMAERDSBHQNGZHTQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:42.966609148 (79h43m18.529932132s) [noise=true , address=104.36.82.20:6468 , piece_id=CCSQEOUXRJC7AXDSH4EASAPPSK7365ZA72OZI56C4NTVL44U46WQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:43.732966458 (79h43m19.296289462s) [noise=true , address=79.127.219.38:4036 , piece_id=SSRGW7SWOVNO4XSRIKLNDXW5F2R46FTOFQJNUAQQ34PILKHX3I4A, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:45.867417843 (79h43m21.430740847s) [noise=true , address=104.36.82.21:6493 , piece_id=GGBVGKVSAWFYTLYPNME4EBY3QAMQFA2MESSDNWAG7WLAD3NCPHKA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:50.615388344 (79h43m26.178711347s) [noise=true , address=104.36.82.53:4146 , piece_id=3R7U2WZVOFQK3YAMMBQFB62JEV4GO72YSSKZHGWUCOLGSHPYWVEQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:51.372429136 (79h43m26.935752124s) [noise=true , address=104.36.82.56:4241 , piece_id=YJESR3E5OCIDCHNDKB4EO5MPKH5IP4WIWP5FBLVF7A4SSWIDMPDQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:54.589773515 (79h43m30.153096515s) [noise=true , address=104.36.82.54:4186 , piece_id=UWTOIEDDR76OM6Y2ACCRIUTAPMMMR7MDZZJ4WKWR5DJ55IQQ67QA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:06:56.923953508 (79h43m32.487276509s) [noise=true , address=104.36.82.55:4204 , piece_id=TII6GSL5AQ65NVLWSTR4FV4Y3KIGOZTH6VKYSFVK3SNAQ4VAM4HA, seg_pos={0 0}, upload=63815]: Done uploading piece. err: <nil> Jun 22 00:07:35.039734848 (79h44m10.603057859s) [noise=true , address=104.36.82.52:4108 , piece_id=CK6YCPRZFTVJNOBD6WJGXIFXJ3P5ZTKGK76CTN4HGJ3GWVYM6XPQ, seg_pos={0 0}, upload=63815]: Done uploading piece. err: ecclient: upload failed (node:12XJJZCUSHUS52YC6jhoPB4ayjsyKZT477PymPj4inkZtdUvheT, address:104.36.82.52:4108): protocol: write tcp 159.223.143.39:34012->104.36.82.52:4108: use of closed network connection; write tcp 159.223.143.39:34012->104.36.82.52:4108: use of closed network connection Jun 22 00:07:35.039830671 (79h44m10.603153681s) [seg_pos={0 0}, upload=63815]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 22 00:07:35.068529987 (79h44m10.631852995s) [upload=63815]: Done uploading segments... Jun 22 00:07:35.068576498 (79h44m10.631899502s) [upload=63815]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.031411142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 12.746723799s FAIL (2025-06-22 00:06:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.747320387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 24.455690684s FAIL (2025-06-22 00:05:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.455730374s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 20.65356017s FAIL (2025-06-22 00:05:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.653591216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 24.532522667s FAIL (2025-06-22 00:04:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.532694333s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 16.304839128s FAIL (2025-06-22 00:03:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.304872443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 18.430073579s FAIL (2025-06-22 00:02:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.430097685s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X.......................................X...................] duration: 32.808240876s FAIL (2025-06-22 00:02:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.808263518s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X.......................................X..................] duration: 20.454871441s FAIL (2025-06-22 00:01:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.454927894s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X.......................................X.................] duration: 14.368283407s FAIL (2025-06-22 00:00:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.368328073s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)