Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 23 out of 60 is failed (threshold=3) [.......X........................X......XXXXXXXXXXXXXXXXXXXXX] duration: 30.026495313s FAIL (2025-07-26 23:22:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.026529339s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 30.026539942s FAIL (2025-07-26 23:22:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 23:21:54.887654984 (1h30m10.66858837s) [upload=1383]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 23:22:18.565715588 (1h30m34.34664897s) [upload=1395]: Starting upload Jul 26 23:22:18.565801804 (1h30m34.346735183s) [upload=1395]: Done starting upload Jul 26 23:22:18.565871378 (1h30m34.346804758s) [upload=1395]: Uploading segments... Jul 26 23:22:18.566111101 (1h30m34.347044488s) [upload=1395]: Got next segment. Inline: false Jul 26 23:22:18.566169255 (1h30m34.347102637s) [upload=1395]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:22:18.566175426 (1h30m34.347108808s) [upload=1395]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:22:18.676953085 (1h30m34.457886484s) [seg_pos={0 0}, upload=1395]: Begin upload segment... Jul 26 23:22:18.677006845 (1h30m34.457940224s) [seg_pos={0 0}, upload=1395]: Done begin upload segment. Jul 26 23:22:18.677041582 (1h30m34.457974988s) [noise=true , address=104.36.82.2:4729 , piece_id=KRD4T373WUGA2NXAT6OJ6AA52PBZRIIIZIPGN2YFQZOILITCTO2Q, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.677062416 (1h30m34.457995802s) [noise=true , address=45.91.116.6:4975 , piece_id=CLF73O5UP4RJU4WAXYWGBY6NC6BAWVIC3JZ4H4HS6FOPEURR3USA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.677452298 (1h30m34.458385685s) [noise=true , address=45.91.116.9:4262 , piece_id=NFNHJSCWMYC72IT3ZPIAFJ76TRDS7W5ETEML7VVNAQBWNCMNO5ZQ, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.677540134 (1h30m34.458473515s) [noise=true , address=45.91.116.10:4083 , piece_id=E5MWNTRIFLRIJFI264QMXFHYZF55K64G4T2ZOB4JYSUONSDXYG7Q, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.677775760 (1h30m34.458709135s) [noise=true , address=77.90.31.14:4662 , piece_id=DOU77WHHHMO4OZHWUGA7F2BEB42J536JI2IAB4NBMKZ6CZDKDYOA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.677883425 (1h30m34.45881681s) [noise=true , address=104.36.82.14:4221 , piece_id=IPQISEAZ2XSCNQDIQNM6GRAOKLCQGZUFB6L75QA2MMIEMZ75DFLQ, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.678076946 (1h30m34.459010321s) [noise=true , address=104.36.82.4:4802 , piece_id=ZUFZAVMBSVYYJQTXIQTACIMPGHGVZUFLETPGMPZEY42GPXC6CI2A, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.678267530 (1h30m34.459200911s) [noise=true , address=104.36.82.9:4007 , piece_id=MG2W3UGJANGX4PXHA67KR5RHVUJ5EUZJXZDZKRVXWM6RMADBDNHA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.678401502 (1h30m34.459334877s) [noise=true , address=104.36.82.16:4293 , piece_id=EBHNEPXIKLB6MJRXKYH2RRJKDUA4PBCIELYDGS2JORPCLHF2PVHA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.678703139 (1h30m34.459636517s) [noise=true , address=104.36.82.2:4732 , piece_id=FNIRCO4URPUS4JYLBAP6V5WMFOI34ZKV4PNVFUM7C5JQ24F674OQ, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.678984081 (1h30m34.459917464s) [noise=true , address=104.36.82.12:4151 , piece_id=SGTWR6MYRNRSNPVLI6E4UGKTKQPSQ7TREG6WYPH53BYHGJPPH7WA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.679373248 (1h30m34.460306624s) [noise=true , address=45.91.116.8:4047 , piece_id=GBCSXHVMJMC7MDGSYKH3A4VZI7UE3VD352HW3TRI64D55JJPOQCA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.679719787 (1h30m34.460653169s) [noise=true , address=45.91.116.4:4939 , piece_id=G63GNEM3DWU2FQD5WEJPVG6LBAVG4US4CDCDU3H3H6PNS3QC2TIA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.680159105 (1h30m34.46109249s) [noise=true , address=104.36.82.6:4874 , piece_id=SD5E3FZBMAPJ3KITOX7BHUD6KGK7XHKOQ6DZQYJ2FBPQCVBSU7GA, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.680605455 (1h30m34.46153883s) [noise=true , address=45.91.116.14:4191 , piece_id=KQVN73L42RDTC2RNAXSHBG3IZYDKGVSYV2PSQD6TF6OP27LXHQ3A, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.680942854 (1h30m34.461876236s) [noise=true , address=45.91.116.5:4298 , piece_id=RJNAE3UYHYRO37A675HIGSYWAIVO4ZD4ZNVFIHJ6KUDE27YBL6GQ, seg_pos={0 0}, upload=1395]: Uploading piece... Jul 26 23:22:18.749684604 (1h30m34.530617979s) [upload=1395]: Next returned nil segment Jul 26 23:22:18.749717216 (1h30m34.530650592s) [upload=1395]: Waiting for error group managing segments... Jul 26 23:22:18.753292333 (1h30m34.534225702s) [noise=true , address=45.91.116.10:4083 , piece_id=E5MWNTRIFLRIJFI264QMXFHYZF55K64G4T2ZOB4JYSUONSDXYG7Q, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:18.753340094 (1h30m34.534273464s) [noise=true , address=45.91.116.8:4047 , piece_id=GBCSXHVMJMC7MDGSYKH3A4VZI7UE3VD352HW3TRI64D55JJPOQCA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:18.753726053 (1h30m34.534659422s) [noise=true , address=45.91.116.4:4939 , piece_id=G63GNEM3DWU2FQD5WEJPVG6LBAVG4US4CDCDU3H3H6PNS3QC2TIA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:18.753990707 (1h30m34.534924076s) [noise=true , address=45.91.116.6:4975 , piece_id=CLF73O5UP4RJU4WAXYWGBY6NC6BAWVIC3JZ4H4HS6FOPEURR3USA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:18.755955656 (1h30m34.536889025s) [noise=true , address=77.90.31.14:4662 , piece_id=DOU77WHHHMO4OZHWUGA7F2BEB42J536JI2IAB4NBMKZ6CZDKDYOA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:18.880443516 (1h30m34.661376884s) [noise=true , address=45.91.116.14:4191 , piece_id=KQVN73L42RDTC2RNAXSHBG3IZYDKGVSYV2PSQD6TF6OP27LXHQ3A, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:19.000935049 (1h30m34.781868418s) [noise=true , address=45.91.116.5:4298 , piece_id=RJNAE3UYHYRO37A675HIGSYWAIVO4ZD4ZNVFIHJ6KUDE27YBL6GQ, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:19.072966588 (1h30m34.853899964s) [noise=true , address=45.91.116.9:4262 , piece_id=NFNHJSCWMYC72IT3ZPIAFJ76TRDS7W5ETEML7VVNAQBWNCMNO5ZQ, seg_pos={0 0}, upload=1395]: Done uploading piece. err: <nil> Jul 26 23:22:48.533008871 (1h31m4.313942248s) [noise=true , address=104.36.82.14:4221 , piece_id=IPQISEAZ2XSCNQDIQNM6GRAOKLCQGZUFB6L75QA2MMIEMZ75DFLQ, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533104583 (1h31m4.314037967s) [noise=true , address=104.36.82.2:4732 , piece_id=FNIRCO4URPUS4JYLBAP6V5WMFOI34ZKV4PNVFUM7C5JQ24F674OQ, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533207069 (1h31m4.31414045s) [noise=true , address=104.36.82.12:4151 , piece_id=SGTWR6MYRNRSNPVLI6E4UGKTKQPSQ7TREG6WYPH53BYHGJPPH7WA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533280800 (1h31m4.314214168s) [noise=true , address=104.36.82.6:4874 , piece_id=SD5E3FZBMAPJ3KITOX7BHUD6KGK7XHKOQ6DZQYJ2FBPQCVBSU7GA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533359742 (1h31m4.314293113s) [noise=true , address=104.36.82.2:4729 , piece_id=KRD4T373WUGA2NXAT6OJ6AA52PBZRIIIZIPGN2YFQZOILITCTO2Q, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533421750 (1h31m4.31435512s) [noise=true , address=104.36.82.4:4802 , piece_id=ZUFZAVMBSVYYJQTXIQTACIMPGHGVZUFLETPGMPZEY42GPXC6CI2A, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533503864 (1h31m4.314437229s) [noise=true , address=104.36.82.16:4293 , piece_id=EBHNEPXIKLB6MJRXKYH2RRJKDUA4PBCIELYDGS2JORPCLHF2PVHA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533565729 (1h31m4.314499092s) [noise=true , address=104.36.82.9:4007 , piece_id=MG2W3UGJANGX4PXHA67KR5RHVUJ5EUZJXZDZKRVXWM6RMADBDNHA, seg_pos={0 0}, upload=1395]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:48376->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:48.533589676 (1h31m4.314523067s) [seg_pos={0 0}, upload=1395]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:22:48.556625151 (1h31m4.337558522s) [upload=1395]: Done uploading segments... Jul 26 23:22:48.556653190 (1h31m4.337586558s) [upload=1395]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.026567198s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........................................XXX.X....XX.X.X.X...] duration: 12.649190372s FAIL (2025-07-26 23:22:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.649211487s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [................X.X..........XX...X...X..XX.X...X..X.......X] duration: 1m0.003043777s FAIL (2025-07-26 23:22:28 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.003081415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [........X........................X......XXXXXXXXXXXXXXXXXXXX] duration: 30.04204282s FAIL (2025-07-26 23:22:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 3) Duration: 30.04207014s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXX] duration: 1m0.036548333s FAIL (2025-07-26 23:22:21 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== VOYIJAF3UBWMAUQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.528190591 (1h29m37.309123971s) [noise=true , address=77.90.31.10:4536 , piece_id=35F4NXUJ3NH7SU27OZAJPU2GWOACHL4AKAX7TNE2PCYZFLXVONHA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.529363038 (1h29m37.310296422s) [noise=true , address=156.146.43.225:4449 , piece_id=DJGEZPD5P64X2WIHWCRXALEHVEGR5NHPGRLMA5AJPDZMOYHSOC6A, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.529914695 (1h29m37.310848077s) [noise=true , address=104.36.82.52:4120 , piece_id=MVPJLSLL65XNXKNEV5PPWGWZZFEX625WSK46RZPGBNX3D6UC372Q, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.530573770 (1h29m37.31150715s) [noise=true , address=104.36.82.45:4934 , piece_id=ZZ6RPWOBUTIUH4HJO2F6BRSAIE4MX25MUAU33RBCGTXM2SWYJ56Q, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.530990045 (1h29m37.31192343s) [noise=true , address=79.127.219.38:4010 , piece_id=QM6SRZEJRYHYU27UHRSZXL3FQNEJTTAJHCEKZ6GAL7YWUFKQCAXA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.531330885 (1h29m37.312264267s) [noise=true , address=77.90.31.11:4579 , piece_id=DNJPW4C4I4NQAOZCMH7G7AU3R2T7D57EUGR6MU3K5UIEHSYVONYA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.531673876 (1h29m37.312607274s) [noise=true , address=104.36.82.49:4995 , piece_id=ZDH47LZTN4KE7UK4IVVCUH4BIKDEP2H3RHOI3QA6PNAGL5LVDRNA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.532146070 (1h29m37.313079452s) [noise=true , address=104.36.82.56:4257 , piece_id=BOOLF7NE75XUAX54VAV6MKZQ3GAXI2VB7WVMKYWMAG3CZQL4VMUQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.532352009 (1h29m37.313285394s) [noise=true , address=45.91.116.12:4230 , piece_id=BHLEAWCYA7RXYPN7RPLB27FWXZUDMJU6UIYX4RKY42OGJ6XF5HOQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.532745552 (1h29m37.313678937s) [noise=true , address=79.127.205.234:4619 , piece_id=5AIP2HHFQK24C5DXMOWDFY6HOZLNHSZIACC4UQW3RIHAYTWLA6GQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.533229557 (1h29m37.314162941s) [noise=true , address=79.127.219.51:4212 , piece_id=UC4GIFF6NJHZXXGICMWBWSSI6X7H7OLPJYNGNNBPXK4VUSDZYNZQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.533822806 (1h29m37.314756189s) [noise=true , address=104.36.82.55:4194 , piece_id=RRVNRK3A7PRECUAEK3IJFENPL4PA5JI7PALTWKLW26A5BOOVHXXQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.534204447 (1h29m37.315137837s) [noise=true , address=104.36.82.47:4548 , piece_id=75CSD3VG3BNCLBHLWDM3YCZLTMHOHNK6QNSJ3XUKVOF6NCVXSR2A, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.534696578 (1h29m37.315629944s) [noise=true , address=79.127.205.233:4640 , piece_id=XBZ5NQ3T6VLXZS6YVR5JVH5S576YWZ7KH7KVS4P3CVGIVA2SHKZA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.535168397 (1h29m37.31610178s) [noise=true , address=104.36.82.12:4181 , piece_id=5SXEVH354HFJ5Y7UKCRQLLWXX7YQL4KUKFLIF77ZS6HXORMCLXAA, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.535841826 (1h29m37.316775212s) [noise=true , address=104.36.82.46:4513 , piece_id=NPWASIZDKGAPNK2N2WR4OMRLVPV64HGOWGITXVWZOBKVKELCJOOQ, seg_pos={0 0}, upload=1381]: Uploading piece... Jul 26 23:21:21.615083883 (1h29m37.396017263s) [upload=1381]: Next returned nil segment Jul 26 23:21:21.615104244 (1h29m37.396037631s) [upload=1381]: Waiting for error group managing segments... Jul 26 23:21:21.656559331 (1h29m37.43749271s) [noise=true , address=77.90.31.11:4579 , piece_id=DNJPW4C4I4NQAOZCMH7G7AU3R2T7D57EUGR6MU3K5UIEHSYVONYA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.111769516 (1h29m37.892702894s) [noise=true , address=79.127.205.233:4640 , piece_id=XBZ5NQ3T6VLXZS6YVR5JVH5S576YWZ7KH7KVS4P3CVGIVA2SHKZA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.152805859 (1h29m37.933739226s) [noise=true , address=79.127.205.246:4285 , piece_id=Y65JV632XOGILLER2DN5YE3HFE2JGBGUO2D5OVOYIJAF3UBWMAUQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.153253815 (1h29m37.934187183s) [noise=true , address=79.127.205.234:4619 , piece_id=5AIP2HHFQK24C5DXMOWDFY6HOZLNHSZIACC4UQW3RIHAYTWLA6GQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.251001850 (1h29m38.031935228s) [noise=true , address=79.127.219.38:4010 , piece_id=QM6SRZEJRYHYU27UHRSZXL3FQNEJTTAJHCEKZ6GAL7YWUFKQCAXA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.259767917 (1h29m38.040701292s) [noise=true , address=45.91.116.12:4230 , piece_id=BHLEAWCYA7RXYPN7RPLB27FWXZUDMJU6UIYX4RKY42OGJ6XF5HOQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:22.504771339 (1h29m38.285704706s) [noise=true , address=156.146.43.225:4449 , piece_id=DJGEZPD5P64X2WIHWCRXALEHVEGR5NHPGRLMA5AJPDZMOYHSOC6A, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:21:23.678988341 (1h29m39.459921711s) [noise=true , address=79.127.219.51:4212 , piece_id=UC4GIFF6NJHZXXGICMWBWSSI6X7H7OLPJYNGNNBPXK4VUSDZYNZQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: <nil> Jul 26 23:22:21.359701382 (1h30m37.140634758s) [noise=true , address=104.36.82.46:4513 , piece_id=NPWASIZDKGAPNK2N2WR4OMRLVPV64HGOWGITXVWZOBKVKELCJOOQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: rpc: dial tcp 104.36.82.46:4513: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.359772236 (1h30m37.140705611s) [noise=true , address=104.36.82.54:4183 , piece_id=SE4MN6GQKP2LIM3J4L7EBBC2MX5LOE2B5FZHQQ4L3Q7563M6Y5FQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: rpc: dial tcp 104.36.82.54:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.359828740 (1h30m37.140762115s) [noise=true , address=104.36.82.45:4934 , piece_id=ZZ6RPWOBUTIUH4HJO2F6BRSAIE4MX25MUAU33RBCGTXM2SWYJ56Q, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:12TpurbQEmcBR8qoHKmAtxJAQinkzi8p5tX4Bn5i3UbDQ3kZHGT, address:104.36.82.45:4934): protocol: rpc: dial tcp 104.36.82.45:4934: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.359875499 (1h30m37.140808873s) [noise=true , address=77.90.31.10:4536 , piece_id=35F4NXUJ3NH7SU27OZAJPU2GWOACHL4AKAX7TNE2PCYZFLXVONHA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360136500 (1h30m37.141069875s) [noise=true , address=104.36.82.49:4995 , piece_id=ZDH47LZTN4KE7UK4IVVCUH4BIKDEP2H3RHOI3QA6PNAGL5LVDRNA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1F5oTP56RsDAA7Yde8hYvq7Lx5tiFDUHoBvPrYnhQiFy2Rko3N, address:104.36.82.49:4995): protocol: rpc: dial tcp 104.36.82.49:4995: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360236170 (1h30m37.141169546s) [noise=true , address=104.36.82.56:4257 , piece_id=BOOLF7NE75XUAX54VAV6MKZQ3GAXI2VB7WVMKYWMAG3CZQL4VMUQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:12iw1KEPBAYhvSachvkigv7bPXfGN5XEugvxGC5YTBV2Muymir3, address:104.36.82.56:4257): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:22:21.360310334 (1h30m37.14124371s) [noise=true , address=104.36.82.12:4181 , piece_id=5SXEVH354HFJ5Y7UKCRQLLWXX7YQL4KUKFLIF77ZS6HXORMCLXAA, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1XH92Sdi4ZiRg1iic5A9Mp8RyKZsk9WfVQHLoQ8t9Vvk2Lyxwh, address:104.36.82.12:4181): protocol: rpc: dial tcp 104.36.82.12:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360357676 (1h30m37.141291064s) [noise=true , address=104.36.82.55:4194 , piece_id=RRVNRK3A7PRECUAEK3IJFENPL4PA5JI7PALTWKLW26A5BOOVHXXQ, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:104.36.82.55:4194): protocol: rpc: dial tcp 104.36.82.55:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360416360 (1h30m37.141349738s) [noise=true , address=104.36.82.47:4548 , piece_id=75CSD3VG3BNCLBHLWDM3YCZLTMHOHNK6QNSJ3XUKVOF6NCVXSR2A, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360432743 (1h30m37.141366119s) [noise=true , address=104.36.82.52:4120 , piece_id=MVPJLSLL65XNXKNEV5PPWGWZZFEX625WSK46RZPGBNX3D6UC372Q, seg_pos={0 0}, upload=1381]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: write tcp 159.223.143.39:42460->104.36.82.52:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:22:21.360455657 (1h30m37.141389096s) [seg_pos={0 0}, upload=1381]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:22:21.394293706 (1h30m37.175227076s) [upload=1381]: Done uploading segments... Jul 26 23:22:21.394319252 (1h30m37.175252616s) [upload=1381]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.0365841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.043710063s FAIL (2025-07-26 23:21:54 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== (1h29m40.656875361s) [upload=1383]: Got next segment. Inline: false Jul 26 23:21:24.875986098 (1h29m40.656919484s) [upload=1383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:21:24.876111779 (1h29m40.657045154s) [upload=1383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:21:24.904134144 (1h29m40.68506753s) [seg_pos={0 0}, upload=1383]: Begin upload segment... Jul 26 23:21:24.904188340 (1h29m40.685121729s) [seg_pos={0 0}, upload=1383]: Done begin upload segment. Jul 26 23:21:24.904217544 (1h29m40.685151046s) [noise=true , address=104.36.82.7:4912 , piece_id=JNUNFLJZQ7WXSP5V7A3PJOEPSVKHBVWUZ7JRSMUGBENQT2Y5KL7Q, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.904475648 (1h29m40.685409029s) [noise=true , address=104.36.82.15:4258 , piece_id=B6RVNLJVLHBKIXPMV7CKCC6WTHFQOLFDGPM37R6IJ76EGFAAN37Q, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.904730497 (1h29m40.68566388s) [noise=true , address=45.91.116.7:4009 , piece_id=VXZG53SE2L3HCHTB3ONECGTCZHUH4I7PF2T7J5PLKU73GVENVRRA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.905041032 (1h29m40.685974422s) [noise=true , address=104.36.82.13:4187 , piece_id=WZBPOFAYZ6TVOKQOUFYYKF3ZAXUGUG6YGQQE2FA4JF5Y5AUIRDHQ, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.905200202 (1h29m40.686133583s) [noise=true , address=104.36.82.10:4077 , piece_id=LH7HSXRP5F3M27OOL22WRDJYQ4D6UU6QWGUC236CLQ52VHI6ACBQ, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.905415045 (1h29m40.68634841s) [noise=true , address=104.36.82.16:4294 , piece_id=RQI6N65VRKGMQQX3PJD43M6CBFWEX2UDQBEO6FF6RVCRNADYGJEQ, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.905661910 (1h29m40.686595293s) [noise=true , address=104.36.82.15:4259 , piece_id=ONBEFHTQVY5KNBDARLL7WNHG43J4FYT2EYHBWN6VQMO53F5SCZJA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.905823721 (1h29m40.686757104s) [noise=true , address=45.91.116.8:4048 , piece_id=53LFZ3LUECBKFYH5T7DBSMQPJBQT2KFTQIYPZTDR2X66BIEMYTHA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.906095107 (1h29m40.687028486s) [noise=true , address=45.91.116.4:4940 , piece_id=6UFARQPESYEZH4DT7A5OHKRXGPYBND7IOB4G2XXKM3YTZB6XHSIQ, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.906224964 (1h29m40.687158342s) [noise=true , address=104.36.82.13:4186 , piece_id=RKOV6YNLKDATTPUOLI6TYIOKWDCLGVG5G63SPFOQCFZNIH5NB7GA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.906521187 (1h29m40.687454569s) [noise=true , address=104.36.82.5:4840 , piece_id=MBPJZEIYDLTTDYRE4I2LAEGV6ITKQ73UWQFOLELVJQ3ZWFQO5S3Q, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.906705673 (1h29m40.687639055s) [noise=true , address=104.36.82.16:4293 , piece_id=AGVINMCCQBK6UF6XVY5SBY7KNB3ISFMDEJL3KKQOCBO3RY6W36HA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.906880616 (1h29m40.687813995s) [noise=true , address=104.36.82.6:4876 , piece_id=6K7WYV26EMCATYFGEMNXSFB6V65WLZXSDDDDLK743GJ4GCGMTYPQ, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.907361949 (1h29m40.688295333s) [noise=true , address=104.36.82.5:4837 , piece_id=W3XRFKCS7HBWFH4PQ4JAXLFBVKS6ADV73F5KEGZUFW3BHNN246IA, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.907537001 (1h29m40.688470389s) [noise=true , address=45.91.116.10:4083 , piece_id=LH7SGQOJ2RPVDLDE7VWVSFGCLSA3LQS4VDFGA5X24JKHBOGM4R4Q, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.907952444 (1h29m40.688885824s) [noise=true , address=104.36.82.6:4874 , piece_id=UTFDFFBSZXICASY4WMKYS7VYDP6433XBHX2MYYZNVG5WH2VAS72A, seg_pos={0 0}, upload=1383]: Uploading piece... Jul 26 23:21:24.987902788 (1h29m40.768836164s) [upload=1383]: Next returned nil segment Jul 26 23:21:24.987921368 (1h29m40.768854751s) [upload=1383]: Waiting for error group managing segments... Jul 26 23:21:24.992228344 (1h29m40.773161716s) [noise=true , address=45.91.116.4:4940 , piece_id=6UFARQPESYEZH4DT7A5OHKRXGPYBND7IOB4G2XXKM3YTZB6XHSIQ, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:24.992520053 (1h29m40.77345342s) [noise=true , address=45.91.116.8:4048 , piece_id=53LFZ3LUECBKFYH5T7DBSMQPJBQT2KFTQIYPZTDR2X66BIEMYTHA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:24.992813246 (1h29m40.773746623s) [noise=true , address=45.91.116.10:4083 , piece_id=LH7SGQOJ2RPVDLDE7VWVSFGCLSA3LQS4VDFGA5X24JKHBOGM4R4Q, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:24.993388771 (1h29m40.774322141s) [noise=true , address=45.91.116.7:4009 , piece_id=VXZG53SE2L3HCHTB3ONECGTCZHUH4I7PF2T7J5PLKU73GVENVRRA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:25.312949486 (1h29m41.093882869s) [noise=true , address=104.36.82.5:4840 , piece_id=MBPJZEIYDLTTDYRE4I2LAEGV6ITKQ73UWQFOLELVJQ3ZWFQO5S3Q, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:25.313201008 (1h29m41.094134376s) [noise=true , address=104.36.82.5:4837 , piece_id=W3XRFKCS7HBWFH4PQ4JAXLFBVKS6ADV73F5KEGZUFW3BHNN246IA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: <nil> Jul 26 23:21:54.845476978 (1h30m10.626410353s) [noise=true , address=104.36.82.13:4186 , piece_id=RKOV6YNLKDATTPUOLI6TYIOKWDCLGVG5G63SPFOQCFZNIH5NB7GA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845538046 (1h30m10.626471422s) [noise=true , address=104.36.82.16:4293 , piece_id=AGVINMCCQBK6UF6XVY5SBY7KNB3ISFMDEJL3KKQOCBO3RY6W36HA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845588026 (1h30m10.626521402s) [noise=true , address=104.36.82.16:4294 , piece_id=RQI6N65VRKGMQQX3PJD43M6CBFWEX2UDQBEO6FF6RVCRNADYGJEQ, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845632454 (1h30m10.626565829s) [noise=true , address=104.36.82.15:4259 , piece_id=ONBEFHTQVY5KNBDARLL7WNHG43J4FYT2EYHBWN6VQMO53F5SCZJA, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845686829 (1h30m10.626620203s) [noise=true , address=104.36.82.6:4874 , piece_id=UTFDFFBSZXICASY4WMKYS7VYDP6433XBHX2MYYZNVG5WH2VAS72A, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:21:54.845765850 (1h30m10.626699226s) [noise=true , address=104.36.82.10:4077 , piece_id=LH7HSXRP5F3M27OOL22WRDJYQ4D6UU6QWGUC236CLQ52VHI6ACBQ, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:35210->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845814715 (1h30m10.62674809s) [noise=true , address=104.36.82.7:4912 , piece_id=JNUNFLJZQ7WXSP5V7A3PJOEPSVKHBVWUZ7JRSMUGBENQT2Y5KL7Q, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845849726 (1h30m10.626783104s) [noise=true , address=104.36.82.15:4258 , piece_id=B6RVNLJVLHBKIXPMV7CKCC6WTHFQOLFDGPM37R6IJ76EGFAAN37Q, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845866706 (1h30m10.626800082s) [noise=true , address=104.36.82.13:4187 , piece_id=WZBPOFAYZ6TVOKQOUFYYKF3ZAXUGUG6YGQQE2FA4JF5Y5AUIRDHQ, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845934911 (1h30m10.626868276s) [noise=true , address=104.36.82.6:4876 , piece_id=6K7WYV26EMCATYFGEMNXSFB6V65WLZXSDDDDLK743GJ4GCGMTYPQ, seg_pos={0 0}, upload=1383]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:54.845962141 (1h30m10.626895526s) [seg_pos={0 0}, upload=1383]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:21:54.887623879 (1h30m10.668557263s) [upload=1383]: Done uploading segments... Jul 26 23:21:54.887654984 (1h30m10.66858837s) [upload=1383]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.043728768s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........................................XXX.X....XX.X.X.X..] duration: 2.601413061s FAIL (2025-07-26 23:21:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.601439352s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 21 out of 60 is failed (threshold=3) [.........X........................X......XXXXXXXXXXXXXXXXXXX] duration: 30.037271059s FAIL (2025-07-26 23:21:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.037298598s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........................................XXX.X....XX.X.X.X.] duration: 18.537861762s FAIL (2025-07-26 23:21:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.537903398s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 30.034629034s FAIL (2025-07-26 23:21:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:20:19.696282789 (1h28m35.47721618s) [upload=1359]: Done uploading segments... Jul 26 23:20:19.696316054 (1h28m35.477249438s) [upload=1359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 23:20:54.780605082 (1h29m10.56153845s) [upload=1375]: Starting upload Jul 26 23:20:54.780685097 (1h29m10.561618473s) [upload=1375]: Done starting upload Jul 26 23:20:54.780873246 (1h29m10.561806619s) [upload=1375]: Uploading segments... Jul 26 23:20:54.780969346 (1h29m10.561902725s) [upload=1375]: Got next segment. Inline: false Jul 26 23:20:54.781023786 (1h29m10.561957151s) [upload=1375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:20:54.781030824 (1h29m10.561964191s) [upload=1375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:20:54.821098068 (1h29m10.60203146s) [seg_pos={0 0}, upload=1375]: Begin upload segment... Jul 26 23:20:54.821186209 (1h29m10.602119594s) [seg_pos={0 0}, upload=1375]: Done begin upload segment. Jul 26 23:20:54.821237776 (1h29m10.602171163s) [noise=true , address=45.91.116.14:4191 , piece_id=RHVNRXOHIFGUHA2XURBFDQESX4RN3LBQPH6Q4JS6VCKWFBWB2P5A, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.821251276 (1h29m10.602184665s) [noise=true , address=45.91.116.4:4939 , piece_id=5UM7Z3J4ZOGSX6F54HMY2F6QIMSDV5LZI5UG5KQEBXNXSN4TH2KA, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.821836688 (1h29m10.602770068s) [noise=true , address=104.36.82.13:4186 , piece_id=LNN4L3K6DZT7PCAARPOBNLYUQCP4BDXPNLTPXQURDHEMS5EXC6YQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.821854541 (1h29m10.602787937s) [noise=true , address=45.91.116.12:4225 , piece_id=4HNJNZTM4CLGNVZDHUVAQQLLRPSLYU3JUPWZPWIC7UO766MANRCQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.822245568 (1h29m10.603178956s) [noise=true , address=104.36.82.15:4258 , piece_id=UGSEOGCFCUPBF7UTFMOYDPSVHMYTCOU3OYEHQKYGENTYQEMOZWGA, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.822370180 (1h29m10.603303567s) [noise=true , address=104.36.82.9:4007 , piece_id=4DGXNTMQFYGGECEJ7MLPT5SQ3PLN6MMA2CIX4OZNVN5AV5DIGEAQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.822642326 (1h29m10.603575712s) [noise=true , address=104.36.82.4:4803 , piece_id=BHWC2RH7U4BNVKWUZBXMKEXKQ456PP7PMQFOA72VWOIITW2ERMNA, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.822815078 (1h29m10.603748458s) [noise=true , address=104.36.82.5:4839 , piece_id=R42PIHSH7AU63AOEOPEKNECCO7JV7DIRZWAHXNCGGIALX3TOIWNA, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.823048692 (1h29m10.603982116s) [noise=true , address=45.91.116.8:4048 , piece_id=UXGDDX5SWKIK337JSNLNX74YQBYMFFDTMQDUDAXK63HSKHKPEQKA, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.823202570 (1h29m10.604135953s) [noise=true , address=45.91.116.8:4045 , piece_id=VFUJUZAY2N6K6PT4VDRUPXH62IWCIY6ND2UCI2M3OZ2DAV6FDZVQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.823448779 (1h29m10.604382207s) [noise=true , address=104.36.82.4:4802 , piece_id=SPU4LB2HZI4HVO6CPICSLKZCSC2MVJWE5NZEAXTSTUNZ2QVIGSLQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.823643543 (1h29m10.604576921s) [noise=true , address=45.91.116.6:4975 , piece_id=KLBSONQWZ3BP255JFMCJ2H3G7ZXWABOYPLX6WMRIGRXZK7YKX6DQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.824369551 (1h29m10.605302936s) [noise=true , address=104.36.82.7:4910 , piece_id=SXGCG2TV3ZB2T67GQVBR3M5WMGHPRAETDHYKDDEONEST35C26ERQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.825017700 (1h29m10.605951088s) [noise=true , address=104.36.82.2:4732 , piece_id=5AMJWB6CQ3HDRWXWGRDKSJCURN7VWYASAZU2U6HJVEFRZGQF3VGQ, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.825476169 (1h29m10.60640955s) [noise=true , address=77.90.31.11:4555 , piece_id=GLYBS57U43IH5R5A4UAFHPF7L5NLHNXJB7MDZMOKOYPXOHDKSS3A, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.825965990 (1h29m10.606899379s) [noise=true , address=45.91.116.7:4011 , piece_id=FTF4HD4SNSGMR5KFO2MKW2ABAXTZ4KBA6AQZTT4ZPTAWZPZ23U7Q, seg_pos={0 0}, upload=1375]: Uploading piece... Jul 26 23:20:54.910664577 (1h29m10.691597945s) [upload=1375]: Next returned nil segment Jul 26 23:20:54.910695762 (1h29m10.691629141s) [upload=1375]: Waiting for error group managing segments... Jul 26 23:20:54.926409737 (1h29m10.707343107s) [noise=true , address=45.91.116.8:4048 , piece_id=UXGDDX5SWKIK337JSNLNX74YQBYMFFDTMQDUDAXK63HSKHKPEQKA, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.926903994 (1h29m10.707837372s) [noise=true , address=45.91.116.6:4975 , piece_id=KLBSONQWZ3BP255JFMCJ2H3G7ZXWABOYPLX6WMRIGRXZK7YKX6DQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.927469880 (1h29m10.70840326s) [noise=true , address=45.91.116.8:4045 , piece_id=VFUJUZAY2N6K6PT4VDRUPXH62IWCIY6ND2UCI2M3OZ2DAV6FDZVQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.928199677 (1h29m10.709133051s) [noise=true , address=45.91.116.4:4939 , piece_id=5UM7Z3J4ZOGSX6F54HMY2F6QIMSDV5LZI5UG5KQEBXNXSN4TH2KA, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.928674369 (1h29m10.709607743s) [noise=true , address=45.91.116.12:4225 , piece_id=4HNJNZTM4CLGNVZDHUVAQQLLRPSLYU3JUPWZPWIC7UO766MANRCQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.929086418 (1h29m10.710019799s) [noise=true , address=45.91.116.7:4011 , piece_id=FTF4HD4SNSGMR5KFO2MKW2ABAXTZ4KBA6AQZTT4ZPTAWZPZ23U7Q, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:54.963009092 (1h29m10.743942468s) [noise=true , address=77.90.31.11:4555 , piece_id=GLYBS57U43IH5R5A4UAFHPF7L5NLHNXJB7MDZMOKOYPXOHDKSS3A, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:20:55.197203553 (1h29m10.978136921s) [noise=true , address=45.91.116.14:4191 , piece_id=RHVNRXOHIFGUHA2XURBFDQESX4RN3LBQPH6Q4JS6VCKWFBWB2P5A, seg_pos={0 0}, upload=1375]: Done uploading piece. err: <nil> Jul 26 23:21:24.745983768 (1h29m40.526917133s) [noise=true , address=104.36.82.5:4839 , piece_id=R42PIHSH7AU63AOEOPEKNECCO7JV7DIRZWAHXNCGGIALX3TOIWNA, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746050223 (1h29m40.526983588s) [noise=true , address=104.36.82.15:4258 , piece_id=UGSEOGCFCUPBF7UTFMOYDPSVHMYTCOU3OYEHQKYGENTYQEMOZWGA, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746102931 (1h29m40.527036296s) [noise=true , address=104.36.82.4:4803 , piece_id=BHWC2RH7U4BNVKWUZBXMKEXKQ456PP7PMQFOA72VWOIITW2ERMNA, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746153911 (1h29m40.527087274s) [noise=true , address=104.36.82.4:4802 , piece_id=SPU4LB2HZI4HVO6CPICSLKZCSC2MVJWE5NZEAXTSTUNZ2QVIGSLQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746211128 (1h29m40.527144503s) [noise=true , address=104.36.82.13:4186 , piece_id=LNN4L3K6DZT7PCAARPOBNLYUQCP4BDXPNLTPXQURDHEMS5EXC6YQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746265059 (1h29m40.527198421s) [noise=true , address=104.36.82.9:4007 , piece_id=4DGXNTMQFYGGECEJ7MLPT5SQ3PLN6MMA2CIX4OZNVN5AV5DIGEAQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:24.746383434 (1h29m40.527316811s) [noise=true , address=104.36.82.7:4910 , piece_id=SXGCG2TV3ZB2T67GQVBR3M5WMGHPRAETDHYKDDEONEST35C26ERQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:21:24.746555461 (1h29m40.527488837s) [noise=true , address=104.36.82.2:4732 , piece_id=5AMJWB6CQ3HDRWXWGRDKSJCURN7VWYASAZU2U6HJVEFRZGQF3VGQ, seg_pos={0 0}, upload=1375]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:21:24.746586951 (1h29m40.527520346s) [seg_pos={0 0}, upload=1375]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:21:24.778006617 (1h29m40.558939985s) [upload=1375]: Done uploading segments... Jul 26 23:21:24.778075045 (1h29m40.55900841s) [upload=1375]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.034733118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXX] duration: 1m0.032975824s FAIL (2025-07-26 23:21:21 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== m37.06439314s) [upload=1366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:20:21.322547368 (1h28m37.103480733s) [seg_pos={0 0}, upload=1366]: Begin upload segment... Jul 26 23:20:21.322592790 (1h28m37.103526157s) [seg_pos={0 0}, upload=1366]: Done begin upload segment. Jul 26 23:20:21.322632798 (1h28m37.103566179s) [noise=true , address=104.36.82.54:4164 , piece_id=CMR5GRD5NZNE4TUABNZIMRD6WDH4WCWHYKZXRRQLMNKRHFAIEGSA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.323122749 (1h28m37.10405613s) [noise=true , address=79.127.219.47:4061 , piece_id=ZYPHOABIF26IHVTA3XFY43AYX2JNRW4DGHFCKT27J42ZBG7ZBMUA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.323260737 (1h28m37.104194124s) [noise=true , address=104.36.82.55:4201 , piece_id=QOU753O2MECZ5BRGQURSR56CKVTBWCUIDNGWBWL4NQALAGYSJK6Q, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.323630846 (1h28m37.104564231s) [noise=true , address=77.90.31.10:4533 , piece_id=EOOAUUFLAU73UDM2DB54APPWPEMQZ2D3SHMOEY4TMSAAZBKONJ2Q, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.323740940 (1h28m37.104674319s) [noise=true , address=104.36.82.41:4767 , piece_id=4CH2NP626LIIQK2LUBB67EYXO4WQ6LT2Z2IMQQILS5DTDLXPVIMQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.324078417 (1h28m37.105011802s) [noise=true , address=104.36.82.47:4521 , piece_id=M46ALQQ2B3F6DMVURKWECLRTQXF4JO2QFDYI5H6B5ODYUDW5LVCA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.324223156 (1h28m37.105156538s) [noise=true , address=79.127.201.215:4355 , piece_id=SIXB7SETQSE6D4A23P4IL27XP2OJ65YITSJODRQHF475YKYAL3PQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.324458344 (1h28m37.105391729s) [noise=true , address=104.36.82.52:4096 , piece_id=6VR3FUBXLPSTRTJRGV2JOHD5O5JK76OOAYBOUF3ZNLPJGLIC2FHA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.324654591 (1h28m37.105587983s) [noise=true , address=104.36.82.46:4490 , piece_id=FM7B3RPVGJAFSBPQBLYSGWZPAU4NOI6LAFYG5X2NVKT2ZIDP37ZQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.324839447 (1h28m37.105772833s) [noise=true , address=64.49.13.170:4721 , piece_id=2UW3CM5JHWK6NQU7KBWVBXYGVF5RWKPH3WH3H7MXNW3OPRJLRMMQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325110176 (1h28m37.10604355s) [noise=true , address=77.90.31.15:4712 , piece_id=CS2J544LZFHBITY7YOH5FG6LWWZFHS6U4GULPUTIGIRI5UNGUYIA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325202354 (1h28m37.106135946s) [noise=true , address=77.90.31.11:4585 , piece_id=CDUWDVDXRQVCB5QR2UW5V2IZ6NSNELF5RPIE4WDWUCEE4PP6BZDA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325510983 (1h28m37.10644436s) [noise=true , address=64.49.13.140:4514 , piece_id=MUHEAX5ICBXCPGTYWPY634ZQZPSDGMLPN6URBGDD6YNRFKJG7ZHQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325584342 (1h28m37.106517725s) [noise=true , address=79.127.219.51:4210 , piece_id=3UAXVVDRK33TPOIWHNUUN7GIVS7XTY4BFMJ5KLENJBPI6QLSJ25A, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325847398 (1h28m37.106780775s) [noise=true , address=79.127.205.234:4603 , piece_id=VJAZYCAMJVZBUIMQWXICWUDLNAI3OS7MGXPAEOHDRCA763APB5CQ, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.325947492 (1h28m37.106880876s) [noise=true , address=77.90.31.18:4809 , piece_id=YVEJ6DSHA7ARPP4MCANZHPSJLDZGZ3KXX3YXRBGEMURALS23KSAA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.326263032 (1h28m37.107196417s) [noise=true , address=79.127.205.245:4232 , piece_id=WS5DHVPBRDHOXCD2BAGQKCMJ6ID2QGBFQK6I2YDCLFWCQ6E5NGVA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.326332449 (1h28m37.107265831s) [noise=true , address=79.127.220.97:4361 , piece_id=WRNCBFABFZH36U4X6RK7RU2E37GFIER2SE3SKS347CLKI52AH3ZA, seg_pos={0 0}, upload=1366]: Uploading piece... Jul 26 23:20:21.405775038 (1h28m37.186708419s) [upload=1366]: Next returned nil segment Jul 26 23:20:21.405806965 (1h28m37.18674035s) [upload=1366]: Waiting for error group managing segments... Jul 26 23:20:21.464127395 (1h28m37.245060761s) [noise=true , address=77.90.31.18:4809 , piece_id=YVEJ6DSHA7ARPP4MCANZHPSJLDZGZ3KXX3YXRBGEMURALS23KSAA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:21.477125315 (1h28m37.258058682s) [noise=true , address=77.90.31.11:4585 , piece_id=CDUWDVDXRQVCB5QR2UW5V2IZ6NSNELF5RPIE4WDWUCEE4PP6BZDA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:21.524459841 (1h28m37.305393211s) [noise=true , address=79.127.220.97:4361 , piece_id=WRNCBFABFZH36U4X6RK7RU2E37GFIER2SE3SKS347CLKI52AH3ZA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:21.931563104 (1h28m37.712496471s) [noise=true , address=79.127.205.234:4603 , piece_id=VJAZYCAMJVZBUIMQWXICWUDLNAI3OS7MGXPAEOHDRCA763APB5CQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:22.030922152 (1h28m37.811855515s) [noise=true , address=64.49.13.170:4721 , piece_id=2UW3CM5JHWK6NQU7KBWVBXYGVF5RWKPH3WH3H7MXNW3OPRJLRMMQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:22.188677637 (1h28m37.969611007s) [noise=true , address=79.127.205.245:4232 , piece_id=WS5DHVPBRDHOXCD2BAGQKCMJ6ID2QGBFQK6I2YDCLFWCQ6E5NGVA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:23.911736832 (1h28m39.692670212s) [noise=true , address=79.127.219.51:4210 , piece_id=3UAXVVDRK33TPOIWHNUUN7GIVS7XTY4BFMJ5KLENJBPI6QLSJ25A, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:25.147147055 (1h28m40.928080421s) [noise=true , address=79.127.219.47:4061 , piece_id=ZYPHOABIF26IHVTA3XFY43AYX2JNRW4DGHFCKT27J42ZBG7ZBMUA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:25.624318759 (1h28m41.405252121s) [noise=true , address=77.90.31.15:4712 , piece_id=CS2J544LZFHBITY7YOH5FG6LWWZFHS6U4GULPUTIGIRI5UNGUYIA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:31.109755674 (1h28m46.890689043s) [noise=true , address=64.49.13.140:4514 , piece_id=MUHEAX5ICBXCPGTYWPY634ZQZPSDGMLPN6URBGDD6YNRFKJG7ZHQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:20:51.564486401 (1h29m7.345419769s) [noise=true , address=79.127.201.215:4355 , piece_id=SIXB7SETQSE6D4A23P4IL27XP2OJ65YITSJODRQHF475YKYAL3PQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: <nil> Jul 26 23:21:21.241425049 (1h29m37.022358425s) [noise=true , address=104.36.82.47:4521 , piece_id=M46ALQQ2B3F6DMVURKWECLRTQXF4JO2QFDYI5H6B5ODYUDW5LVCA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD, address:104.36.82.47:4521): protocol: write tcp 159.223.143.39:46802->104.36.82.47:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241508504 (1h29m37.022441881s) [noise=true , address=104.36.82.54:4164 , piece_id=CMR5GRD5NZNE4TUABNZIMRD6WDH4WCWHYKZXRRQLMNKRHFAIEGSA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:1gUXWffNccBoXWftiMWHHSTeEpA5zrd2cHz4uXX5j75GeEK9s2, address:104.36.82.54:4164): protocol: rpc: dial tcp 104.36.82.54:4164: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241583453 (1h29m37.022516829s) [noise=true , address=104.36.82.55:4201 , piece_id=QOU753O2MECZ5BRGQURSR56CKVTBWCUIDNGWBWL4NQALAGYSJK6Q, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:21:21.241657158 (1h29m37.022590535s) [noise=true , address=77.90.31.10:4533 , piece_id=EOOAUUFLAU73UDM2DB54APPWPEMQZ2D3SHMOEY4TMSAAZBKONJ2Q, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: rpc: dial tcp 77.90.31.10:4533: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241737771 (1h29m37.022671147s) [noise=true , address=104.36.82.41:4767 , piece_id=4CH2NP626LIIQK2LUBB67EYXO4WQ6LT2Z2IMQQILS5DTDLXPVIMQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug, address:104.36.82.41:4767): protocol: rpc: dial tcp 104.36.82.41:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241757118 (1h29m37.022690487s) [noise=true , address=104.36.82.52:4096 , piece_id=6VR3FUBXLPSTRTJRGV2JOHD5O5JK76OOAYBOUF3ZNLPJGLIC2FHA, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: rpc: dial tcp 104.36.82.52:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241808976 (1h29m37.022742344s) [noise=true , address=104.36.82.46:4490 , piece_id=FM7B3RPVGJAFSBPQBLYSGWZPAU4NOI6LAFYG5X2NVKT2ZIDP37ZQ, seg_pos={0 0}, upload=1366]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: write tcp 159.223.143.39:52118->104.36.82.46:4490: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:21:21.241852902 (1h29m37.022786271s) [seg_pos={0 0}, upload=1366]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:21:21.271746563 (1h29m37.052679932s) [upload=1366]: Done uploading segments... Jul 26 23:21:21.271775390 (1h29m37.052708754s) [upload=1366]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.03300394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 20 out of 60 is failed (threshold=3) [..........X........................X......XXXXXXXXXXXXXXXXXX] duration: 30.03165876s FAIL (2025-07-26 23:21:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.031683291s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........................................XXX.X....XX.X.X.X] duration: 30.001109198s FAIL (2025-07-26 23:21:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001135816s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.................X.X..........XX...X...X..XX.X...X..X.......] duration: 16.616612562s FAIL (2025-07-26 23:20:44 UTC) context deadline exceeded Duration: 16.616661453s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X.................X.X..........XX...X...X..XX.X...X..X......] duration: 44.472076741s FAIL (2025-07-26 23:20:27 UTC) context deadline exceeded Duration: 44.47209839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 30.03068359s FAIL (2025-07-26 23:20:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== [upload=1359]: Got next segment. Inline: false Jul 26 23:19:49.705179586 (1h28m5.486112962s) [upload=1359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:19:49.705186345 (1h28m5.486119722s) [upload=1359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:19:49.737216056 (1h28m5.518149436s) [seg_pos={0 0}, upload=1359]: Begin upload segment... Jul 26 23:19:49.737324400 (1h28m5.518257789s) [noise=true , address=45.91.116.4:4937 , piece_id=32GPWGCNLK53GJJEGCHD4Y7IHBLCU5HEJWTUHGZFWQM5R5I4HTXA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.739175569 (1h28m5.520108958s) [seg_pos={0 0}, upload=1359]: Done begin upload segment. Jul 26 23:19:49.739262330 (1h28m5.520195711s) [noise=true , address=104.36.82.3:4765 , piece_id=AEWI22F3IXJV5XATZK4EZ5P3WSI637TQMOTKXPNULX3DMJOYJUYQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.740128738 (1h28m5.521062126s) [noise=true , address=104.36.82.14:4223 , piece_id=ZV5AKCPNALPANDEG4BZQH5QKGZ72JSPA4ZGTE4SRLWF55IY6SQ6Q, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.740738555 (1h28m5.521671936s) [noise=true , address=45.91.116.10:4081 , piece_id=OGOHUX5VKB27Y2MSGEMQX56SBV5FRG2MFRJ6D3ELOLYIWJ4TKVUA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.741476198 (1h28m5.522409581s) [noise=true , address=77.90.31.14:4661 , piece_id=Z7NP3MXFCD6BG3WDYLH5QAHZUDBXO64VGI3ZJK4L63DYNR4EHWNA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.742098185 (1h28m5.523031567s) [noise=true , address=104.36.82.8:4971 , piece_id=USF5JOGZMJ3SM5KCPYL3MLBRYGQIHHR3UPCHOVUHYRQPY7J7T32A, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.742604736 (1h28m5.523538123s) [noise=true , address=104.36.82.13:4185 , piece_id=YKJBH6MN36IFR4VUQP2O3GQLFUQAY6BRAC6JBSAEYU4RQNQ7D7JQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.743544502 (1h28m5.524477899s) [noise=true , address=45.91.116.13:4155 , piece_id=XBX54JV7XYGKQNKYSC3XD4M7OTI62WORHGJPEQZCR5J4TLHEPXPQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.744422133 (1h28m5.525355532s) [noise=true , address=104.36.82.9:4005 , piece_id=NF46AUDIQPJNS24YMGSQGZR4WOVZBJG3UAO24M4TZA3IX52BXLVA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.744992808 (1h28m5.525926196s) [noise=true , address=104.36.82.6:4875 , piece_id=P4FCC2AHDJKUCVJFATVZMA4RVUNKULSITDTTQN3VOIDUH4V4R7NA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.746722662 (1h28m5.527656052s) [noise=true , address=104.36.82.11:4114 , piece_id=TSGETJOBEYYHHPMZIPMVXGFOZ2XZXWIKF3YSKAQA3MJPN6PYH6YQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.747199309 (1h28m5.528132696s) [noise=true , address=104.36.82.5:4837 , piece_id=RARFXIZKLTG6FOFWAFYESPJA5BOQEVHY5YM4G54YFUXF67Z6FNBA, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.747547213 (1h28m5.528480592s) [noise=true , address=45.91.116.13:4154 , piece_id=X4ZALKZ2DYEJ4NQCPSC7KUV7WIDESFURJHZ5WDELM5NGYWGOYWRQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.747926081 (1h28m5.528859461s) [noise=true , address=45.91.116.5:4299 , piece_id=B67XJC3Y74ZVW6G344CELCUFGB72E64H6GI5HQ2UALWSJ7AACV4A, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.748331977 (1h28m5.52926537s) [noise=true , address=104.36.82.10:4079 , piece_id=EUQTUOWX66CFSX2ZGOBAEPYDJ7I3IRJXMA4EUX2EMEFUX6LIUDJQ, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.748745773 (1h28m5.52967916s) [noise=true , address=104.36.82.4:4803 , piece_id=DXNYZOV6KI7BDE34WGVDGFHWJMT3YZG3AXLNPH342VW6W6FAP76A, seg_pos={0 0}, upload=1359]: Uploading piece... Jul 26 23:19:49.838319936 (1h28m5.619253314s) [upload=1359]: Next returned nil segment Jul 26 23:19:49.838329763 (1h28m5.619263128s) [upload=1359]: Waiting for error group managing segments... Jul 26 23:19:49.842535778 (1h28m5.623469145s) [noise=true , address=45.91.116.13:4155 , piece_id=XBX54JV7XYGKQNKYSC3XD4M7OTI62WORHGJPEQZCR5J4TLHEPXPQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:19:49.842933898 (1h28m5.623867261s) [noise=true , address=45.91.116.10:4081 , piece_id=OGOHUX5VKB27Y2MSGEMQX56SBV5FRG2MFRJ6D3ELOLYIWJ4TKVUA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:19:49.843423189 (1h28m5.624356566s) [noise=true , address=45.91.116.5:4299 , piece_id=B67XJC3Y74ZVW6G344CELCUFGB72E64H6GI5HQ2UALWSJ7AACV4A, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:19:49.843876736 (1h28m5.624810098s) [noise=true , address=45.91.116.4:4937 , piece_id=32GPWGCNLK53GJJEGCHD4Y7IHBLCU5HEJWTUHGZFWQM5R5I4HTXA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:19:49.844397457 (1h28m5.625330839s) [noise=true , address=77.90.31.14:4661 , piece_id=Z7NP3MXFCD6BG3WDYLH5QAHZUDBXO64VGI3ZJK4L63DYNR4EHWNA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:19:49.845034068 (1h28m5.625967429s) [noise=true , address=45.91.116.13:4154 , piece_id=X4ZALKZ2DYEJ4NQCPSC7KUV7WIDESFURJHZ5WDELM5NGYWGOYWRQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: <nil> Jul 26 23:20:19.668998314 (1h28m35.449931693s) [noise=true , address=104.36.82.13:4185 , piece_id=YKJBH6MN36IFR4VUQP2O3GQLFUQAY6BRAC6JBSAEYU4RQNQ7D7JQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669096474 (1h28m35.450029862s) [noise=true , address=104.36.82.9:4005 , piece_id=NF46AUDIQPJNS24YMGSQGZR4WOVZBJG3UAO24M4TZA3IX52BXLVA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669186252 (1h28m35.45011964s) [noise=true , address=104.36.82.10:4079 , piece_id=EUQTUOWX66CFSX2ZGOBAEPYDJ7I3IRJXMA4EUX2EMEFUX6LIUDJQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669266337 (1h28m35.4501997s) [noise=true , address=104.36.82.14:4223 , piece_id=ZV5AKCPNALPANDEG4BZQH5QKGZ72JSPA4ZGTE4SRLWF55IY6SQ6Q, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669356614 (1h28m35.450289993s) [noise=true , address=104.36.82.8:4971 , piece_id=USF5JOGZMJ3SM5KCPYL3MLBRYGQIHHR3UPCHOVUHYRQPY7J7T32A, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669427226 (1h28m35.450360606s) [noise=true , address=104.36.82.6:4875 , piece_id=P4FCC2AHDJKUCVJFATVZMA4RVUNKULSITDTTQN3VOIDUH4V4R7NA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669515013 (1h28m35.450448391s) [noise=true , address=104.36.82.11:4114 , piece_id=TSGETJOBEYYHHPMZIPMVXGFOZ2XZXWIKF3YSKAQA3MJPN6PYH6YQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669580172 (1h28m35.450513551s) [noise=true , address=104.36.82.4:4803 , piece_id=DXNYZOV6KI7BDE34WGVDGFHWJMT3YZG3AXLNPH342VW6W6FAP76A, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669642835 (1h28m35.450576218s) [noise=true , address=104.36.82.3:4765 , piece_id=AEWI22F3IXJV5XATZK4EZ5P3WSI637TQMOTKXPNULX3DMJOYJUYQ, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669710891 (1h28m35.450644274s) [noise=true , address=104.36.82.5:4837 , piece_id=RARFXIZKLTG6FOFWAFYESPJA5BOQEVHY5YM4G54YFUXF67Z6FNBA, seg_pos={0 0}, upload=1359]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 159.223.143.39:48088->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:19.669734180 (1h28m35.450667559s) [seg_pos={0 0}, upload=1359]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:20:19.696282789 (1h28m35.47721618s) [upload=1359]: Done uploading segments... Jul 26 23:20:19.696316054 (1h28m35.477249438s) [upload=1359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.030709815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXX] duration: 1m0.027311101s FAIL (2025-07-26 23:20:18 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== ce... Jul 26 23:19:18.135975537 (1h27m33.916908922s) [noise=true , address=79.127.219.47:4048 , piece_id=P2K46E5XGOIYAFSHSRJF3JY6HSPKHED6J764BMLUM7WNWKBX7POA, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.137137960 (1h27m33.918071354s) [noise=true , address=45.91.116.9:4267 , piece_id=LVGE5PKTAQXWUKRIEKPDT7YT4CNCQXPKAITMUR4CVDKR4CLLIOFQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.137513910 (1h27m33.918447295s) [noise=true , address=104.36.82.52:4103 , piece_id=NDEAO2NKTJSZVSSJPDYBX7DIAW5FXKNPW465M6P3FPK27ERWJVQA, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.139192427 (1h27m33.920125811s) [noise=true , address=104.36.82.12:4159 , piece_id=X6SSZ222NHB5K4T45H3YLTAF3N35R6DZCWZCRNWHVF7QPIGYXWWQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.139623129 (1h27m33.920556507s) [noise=true , address=77.90.31.17:4791 , piece_id=DBZUJY3YKUU2PFQLSSHFKMY74Z6MTFBH7R6YGL4N7B5KXZA3WKSA, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.139911553 (1h27m33.920844931s) [noise=true , address=104.36.82.56:4244 , piece_id=2VJ34FX62U5PKA7KQG7LJXEEOJSGWIGU77CYHS7GBHYTWA4Q6JEQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.140405210 (1h27m33.921338591s) [noise=true , address=104.36.82.46:4508 , piece_id=OXUNIGFQJUAVCWGU5QVYCDAJGVI5LAN22YIZU2VEX6Y737YJBMZQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.140790322 (1h27m33.921723814s) [noise=true , address=45.91.116.14:4206 , piece_id=ABMV2VQWYVR3FYW34RFGL4F42BLY7C2PXACAWYKNBNSTVRPSYUXQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.141277180 (1h27m33.922210566s) [noise=true , address=45.91.116.9:4288 , piece_id=XOL24QH2GGVV44AAX47WANUU3Z6NALHYWMROFEZEAGT2MI32ICLQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.141569335 (1h27m33.922502712s) [noise=true , address=64.49.13.138:4482 , piece_id=W7BRVGGVKCQZ46NBWOJ62MQ7ZVHVECKP2I5DPFUQOZ2QVCFBWPOQ, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.142943189 (1h27m33.923876568s) [noise=true , address=77.90.31.11:4564 , piece_id=HYWRCLNSHRNGAI5VQVZDHTOSTWYYU2ADQHC4LZWBNP2QZRT3QILA, seg_pos={0 0}, upload=1351]: Uploading piece... Jul 26 23:19:18.207820665 (1h27m33.988754048s) [upload=1351]: Next returned nil segment Jul 26 23:19:18.207842675 (1h27m33.988776061s) [upload=1351]: Waiting for error group managing segments... Jul 26 23:19:18.280278539 (1h27m34.061211907s) [noise=true , address=77.90.31.11:4564 , piece_id=HYWRCLNSHRNGAI5VQVZDHTOSTWYYU2ADQHC4LZWBNP2QZRT3QILA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: <nil> Jul 26 23:19:19.069249385 (1h27m34.850182755s) [noise=true , address=64.49.13.138:4482 , piece_id=W7BRVGGVKCQZ46NBWOJ62MQ7ZVHVECKP2I5DPFUQOZ2QVCFBWPOQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:19:20.402617511 (1h27m36.183550897s) [noise=true , address=45.91.116.14:4206 , piece_id=ABMV2VQWYVR3FYW34RFGL4F42BLY7C2PXACAWYKNBNSTVRPSYUXQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:19:21.183947592 (1h27m36.96488096s) [noise=true , address=79.127.219.47:4048 , piece_id=P2K46E5XGOIYAFSHSRJF3JY6HSPKHED6J764BMLUM7WNWKBX7POA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: <nil> Jul 26 23:19:23.305968756 (1h27m39.086902122s) [noise=true , address=45.91.116.9:4267 , piece_id=LVGE5PKTAQXWUKRIEKPDT7YT4CNCQXPKAITMUR4CVDKR4CLLIOFQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: <nil> Jul 26 23:19:24.350400852 (1h27m40.131334215s) [noise=true , address=45.91.116.9:4288 , piece_id=XOL24QH2GGVV44AAX47WANUU3Z6NALHYWMROFEZEAGT2MI32ICLQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: <nil> Jul 26 23:20:18.058658937 (1h28m33.839592318s) [noise=true , address=77.90.31.19:4844 , piece_id=Z5GK5IM35AKKL52JWQTT3GU5FR3L7LX4IKIBCPD3LRT3EMBVNBAQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:20:18.058923493 (1h28m33.839856879s) [noise=true , address=104.36.82.46:4508 , piece_id=OXUNIGFQJUAVCWGU5QVYCDAJGVI5LAN22YIZU2VEX6Y737YJBMZQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: rpc: dial tcp 104.36.82.46:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.058994842 (1h28m33.839928229s) [noise=true , address=104.36.82.47:4552 , piece_id=FULWZTWXU44ZZWTW76JDV25PGHTPJ7FA5NE4ZXY62OD6DLC77H4Q, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059093724 (1h28m33.840027091s) [noise=true , address=104.36.82.7:4922 , piece_id=ZP4XY4QZCOKISAC4ZU6GHM3PCJJBGPBVGO2EINHVQ4RSJPJCACMQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12KrMgT4Pwj9UFE88q6tt65D7o6q68NhsmEXmifNmvzJ6eEkm8R, address:104.36.82.7:4922): protocol: write tcp 159.223.143.39:51316->104.36.82.7:4922: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059347392 (1h28m33.840280773s) [noise=true , address=104.36.82.54:4158 , piece_id=XGB5B7N5R52VNDERB5UAEI5MLQTY54ZESM337E2WHV6FCYCAKPQA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:104.36.82.54:4158): protocol: write tcp 159.223.143.39:54914->104.36.82.54:4158: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059429669 (1h28m33.840363038s) [noise=true , address=104.36.82.52:4103 , piece_id=NDEAO2NKTJSZVSSJPDYBX7DIAW5FXKNPW465M6P3FPK27ERWJVQA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: write tcp 159.223.143.39:45784->104.36.82.52:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059509324 (1h28m33.840442701s) [noise=true , address=77.90.31.17:4791 , piece_id=DBZUJY3YKUU2PFQLSSHFKMY74Z6MTFBH7R6YGL4N7B5KXZA3WKSA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: write tcp 159.223.143.39:52948->77.90.31.17:4791: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059650251 (1h28m33.840583629s) [noise=true , address=104.36.82.55:4206 , piece_id=EPMP6WI5FKTXEOFMU4G2C5WOXOSDNU5QPOCVKZ7JQIWLRM5J7PIA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12Mp4ZzYgrP3ZSXDeHbSTXaAJMPgGKUqVjdNtyfeKE5qBMAzBke, address:104.36.82.55:4206): protocol: write tcp 159.223.143.39:56182->104.36.82.55:4206: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059714220 (1h28m33.840647585s) [noise=true , address=104.36.82.53:4147 , piece_id=6OTNFOAJWQYOLRAXPUZEZUMT2H7IQ6EET54L5J3ZCRZUSHAMGJZA, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: write tcp 159.223.143.39:58138->104.36.82.53:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059776542 (1h28m33.840709921s) [noise=true , address=104.36.82.12:4159 , piece_id=X6SSZ222NHB5K4T45H3YLTAF3N35R6DZCWZCRNWHVF7QPIGYXWWQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1taQkf86SDCwKJaXVcfu1YcSNz8SfpAtChruvcZVFfy55qZdtX, address:104.36.82.12:4159): protocol: write tcp 159.223.143.39:59372->104.36.82.12:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059787757 (1h28m33.840721141s) [noise=true , address=104.36.82.56:4244 , piece_id=2VJ34FX62U5PKA7KQG7LJXEEOJSGWIGU77CYHS7GBHYTWA4Q6JEQ, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:12S1wh3PFQLPosjWmPs5iLfm4S5J6abXvoCfkhYz3KdvXCTaPRu, address:104.36.82.56:4244): protocol: write tcp 159.223.143.39:46874->104.36.82.56:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059827230 (1h28m33.840760612s) [noise=true , address=77.90.31.16:4761 , piece_id=M7D5SOI7JKUKFFNCOMTKMZN2DCA6QSQAP6BPIUW3NTDLPNCEGS2Q, seg_pos={0 0}, upload=1351]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:48900->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:20:18.059860232 (1h28m33.84079362s) [seg_pos={0 0}, upload=1351]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 23:20:18.083219670 (1h28m33.86415304s) [upload=1351]: Done uploading segments... Jul 26 23:20:18.083280907 (1h28m33.864214287s) [upload=1351]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.027329753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 19 out of 60 is failed (threshold=3) [...........X........................X......XXXXXXXXXXXXXXXXX] duration: 30.031355956s FAIL (2025-07-26 23:20:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.031386749s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................................XXX.X....XX.X.X.] duration: 23.880314367s FAIL (2025-07-26 23:19:50 UTC) orbiter warning: context deadline exceeded Duration: 23.880345791s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [XX.................X.X..........XX...X...X..XX.X...X..X.....] duration: 55.627074274s FAIL (2025-07-26 23:19:43 UTC) context deadline exceeded Duration: 55.627110065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 18 out of 60 is failed (threshold=3) [............X........................X......XXXXXXXXXXXXXXXX] duration: 30.035692724s FAIL (2025-07-26 23:19:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.035719191s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 30.045171549s FAIL (2025-07-26 23:19:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== Jul 26 23:19:02.159755103 (1h27m17.940688492s) [noise=true , address=104.36.82.12:4150 , piece_id=FI3GLMHJP7HZZJX3XXVC2E3GS6IUXDZNLYGSM4MSKH2F64YNMEXA, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.160279772 (1h27m17.941213163s) [noise=true , address=104.36.82.6:4876 , piece_id=KR6TJ2HCFCXTJISM6UPV7JUNSBROCDCKMTOSOFCRDJHPUPSAFMLQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.161927848 (1h27m17.942861233s) [noise=true , address=104.36.82.6:4874 , piece_id=MVSAK5VFA2G3KNI2P6UCDD4LOIURKXS64WG22366TH5ILGIICKMQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.162315475 (1h27m17.943248858s) [noise=true , address=104.36.82.4:4802 , piece_id=NBH4NEELW65F4QNPEK7HQZ54HTSPKZIN4OPFRJVBA2WESXKY6Q3Q, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.162713925 (1h27m17.943647313s) [noise=true , address=104.36.82.10:4077 , piece_id=LFVEQNFIIGJKSSTQXMEI3ZXWFUEISERM44XRRNTR5SS3OAA2HZBQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.163137239 (1h27m17.944070634s) [noise=true , address=104.36.82.3:4767 , piece_id=PYJ32GWWGMZ2M2N3EXHI2CLQIBH7Y7PPGHSOVPU2DSXEDQU3T3AA, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.163597043 (1h27m17.944530432s) [noise=true , address=45.91.116.7:4010 , piece_id=NVW7C7WAESRXQHLRMSUOHN4VGLQ4JUU4ONZDKYA74HRWNPJ77LRA, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.164025355 (1h27m17.944958745s) [noise=true , address=77.90.31.17:4771 , piece_id=4W7HIAT4QT6AI25QFB32ZTE5G7QVBXGEJWFCKQAEXMWODF2BWHHQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.164464623 (1h27m17.945397997s) [noise=true , address=104.36.82.5:4837 , piece_id=4GM4QNJ4NDEWV6X663ZIXKWTNHABY5NVCW4CSE522OF2OIU3LX6Q, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.164960445 (1h27m17.945893835s) [noise=true , address=104.36.82.8:4972 , piece_id=37M6QNQH6ZB5KIRIJF52JGP7XTQVNI3ZCW7OHF2GTGLIMRFMBALQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.165341867 (1h27m17.946275257s) [noise=true , address=45.91.116.10:4082 , piece_id=B4EMTRIN53U6G34AW46P6CE67M6A3JJSVMBYDPI3JKWBN32YYAVA, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.165747422 (1h27m17.94668081s) [noise=true , address=45.91.116.6:4975 , piece_id=TCZUQCESMM65HGJH3ARHRLVDCCWK2P3LUQJPPRJ2SCPVHOCLWJMQ, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.166129531 (1h27m17.947062921s) [noise=true , address=77.90.31.10:4518 , piece_id=MSBETVWMIRSWRNMIP73TA3MPBSNMTUBEOI2GG4JNTROCJDBE5SKA, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.178017909 (1h27m17.9589513s) [noise=true , address=104.36.82.13:4185 , piece_id=TVHBKI2EHIGUTRHZ2YBA4B75KMAP46TUSCJYSKLGQIHOZWJLRN4A, seg_pos={0 0}, upload=1346]: Uploading piece... Jul 26 23:19:02.274565238 (1h27m18.055498628s) [upload=1346]: Next returned nil segment Jul 26 23:19:02.274595985 (1h27m18.055529374s) [upload=1346]: Waiting for error group managing segments... Jul 26 23:19:02.280814802 (1h27m18.061748183s) [noise=true , address=45.91.116.7:4010 , piece_id=NVW7C7WAESRXQHLRMSUOHN4VGLQ4JUU4ONZDKYA74HRWNPJ77LRA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: <nil> Jul 26 23:19:02.281542924 (1h27m18.062476311s) [noise=true , address=45.91.116.10:4082 , piece_id=B4EMTRIN53U6G34AW46P6CE67M6A3JJSVMBYDPI3JKWBN32YYAVA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: <nil> Jul 26 23:19:02.283976464 (1h27m18.064909847s) [noise=true , address=45.91.116.6:4975 , piece_id=TCZUQCESMM65HGJH3ARHRLVDCCWK2P3LUQJPPRJ2SCPVHOCLWJMQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: <nil> Jul 26 23:19:32.077949022 (1h27m47.858882402s) [noise=true , address=104.36.82.9:4007 , piece_id=XNHF2LWRAOPCMRYWTDCAAGMTHKF5NWGJHEDIH2AXJKBZTUHA6HKA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:50642->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078061043 (1h27m47.858994464s) [noise=true , address=104.36.82.10:4077 , piece_id=LFVEQNFIIGJKSSTQXMEI3ZXWFUEISERM44XRRNTR5SS3OAA2HZBQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:58876->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078180247 (1h27m47.859113627s) [noise=true , address=104.36.82.6:4876 , piece_id=KR6TJ2HCFCXTJISM6UPV7JUNSBROCDCKMTOSOFCRDJHPUPSAFMLQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:51466->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078265362 (1h27m47.85919875s) [noise=true , address=104.36.82.13:4185 , piece_id=TVHBKI2EHIGUTRHZ2YBA4B75KMAP46TUSCJYSKLGQIHOZWJLRN4A, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:53170->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078339117 (1h27m47.859272487s) [noise=true , address=104.36.82.7:4909 , piece_id=P34DSKK7D6BLWPRW5YYUHFX2H4OG55JJUEBGWO2Y4SAT6NR7GZRQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 159.223.143.39:38784->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078634690 (1h27m47.859568074s) [noise=true , address=104.36.82.6:4874 , piece_id=MVSAK5VFA2G3KNI2P6UCDD4LOIURKXS64WG22366TH5ILGIICKMQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078667658 (1h27m47.859601041s) [noise=true , address=104.36.82.3:4767 , piece_id=PYJ32GWWGMZ2M2N3EXHI2CLQIBH7Y7PPGHSOVPU2DSXEDQU3T3AA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078763314 (1h27m47.859696696s) [noise=true , address=104.36.82.4:4802 , piece_id=NBH4NEELW65F4QNPEK7HQZ54HTSPKZIN4OPFRJVBA2WESXKY6Q3Q, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078833065 (1h27m47.859766446s) [noise=true , address=104.36.82.8:4972 , piece_id=37M6QNQH6ZB5KIRIJF52JGP7XTQVNI3ZCW7OHF2GTGLIMRFMBALQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078870910 (1h27m47.859804295s) [noise=true , address=77.90.31.17:4771 , piece_id=4W7HIAT4QT6AI25QFB32ZTE5G7QVBXGEJWFCKQAEXMWODF2BWHHQ, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078905195 (1h27m47.859838558s) [noise=true , address=104.36.82.12:4150 , piece_id=FI3GLMHJP7HZZJX3XXVC2E3GS6IUXDZNLYGSM4MSKH2F64YNMEXA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:55474->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078929978 (1h27m47.85986335s) [noise=true , address=104.36.82.5:4837 , piece_id=4GM4QNJ4NDEWV6X663ZIXKWTNHABY5NVCW4CSE522OF2OIU3LX6Q, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.078963908 (1h27m47.859897293s) [noise=true , address=77.90.31.10:4518 , piece_id=MSBETVWMIRSWRNMIP73TA3MPBSNMTUBEOI2GG4JNTROCJDBE5SKA, seg_pos={0 0}, upload=1346]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:32.079013476 (1h27m47.859946866s) [seg_pos={0 0}, upload=1346]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Jul 26 23:19:32.118793433 (1h27m47.899726822s) [upload=1346]: Done uploading segments... Jul 26 23:19:32.118867751 (1h27m47.899801128s) [upload=1346]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 30.045197889s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXX] duration: 1m0.030011889s FAIL (2025-07-26 23:19:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 18:17.971724315 (1h26m33.752657684s) [upload=1336]: Uploading segments... Jul 26 23:18:17.972298458 (1h26m33.753231828s) [upload=1336]: Got next segment. Inline: false Jul 26 23:18:17.972377867 (1h26m33.753311231s) [upload=1336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:18:17.972384372 (1h26m33.75331775s) [upload=1336]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:18:18.011255720 (1h26m33.792189092s) [seg_pos={0 0}, upload=1336]: Begin upload segment... Jul 26 23:18:18.011308406 (1h26m33.792241772s) [seg_pos={0 0}, upload=1336]: Done begin upload segment. Jul 26 23:18:18.011351967 (1h26m33.792285334s) [noise=true , address=79.127.205.233:4653 , piece_id=SB7L2LSSQKE3XTW7IKQQYJI4KRFDFRVQZV7LXEOCKZQCKGUSU6OQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.011811757 (1h26m33.792745139s) [noise=true , address=79.127.205.245:4225 , piece_id=MDPVVKF3JJJEJJRAP645ZBVV2LKDDIGOF2L4WJBNRTAMLT7NAJEA, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.012328138 (1h26m33.79326152s) [noise=true , address=104.36.82.54:4159 , piece_id=A3OBET2XDW7WKMLL4XCVQZ5WBVE633MY3U72LRCIWLFISREGPTZQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.012868297 (1h26m33.793801675s) [noise=true , address=104.36.82.53:4129 , piece_id=2J3SPS2TCSSKWIV22J2DWMF65LNFIU4FPKU7VMSV3RCADJWBYMSQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.013183201 (1h26m33.794116583s) [noise=true , address=77.90.31.13:4638 , piece_id=CEBNAK2FPBS44DMBK4MVYYBTYNIYCJKBR6L3X3R2QXJPWQ7SWKMA, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.013544174 (1h26m33.794477554s) [noise=true , address=79.127.201.215:4357 , piece_id=UAG3CLZA44VTWYFB4B74CG7I4U6JQFIEDAJBWC4AVROWYN7Y7W4Q, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.013877533 (1h26m33.794810915s) [noise=true , address=45.91.116.11:4151 , piece_id=37N7LUHOAW4PHDVEFRNIEQX4376CF3WEVVXINEE7CZKYLAMMN4RQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.014191229 (1h26m33.795124611s) [noise=true , address=104.36.82.46:4506 , piece_id=RROKRSO5GHA6NKRC7ZAIV6K4BUSIQCCA6XMEDC26WITGRFI35ANA, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.014598985 (1h26m33.795532368s) [noise=true , address=77.90.31.21:4902 , piece_id=WEJPL4MORPE3WIQC44QD4KB2H3YU7TWW2MYBPBLX4Y2IZBAZI27Q, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.017833696 (1h26m33.798767084s) [noise=true , address=45.91.116.14:4194 , piece_id=SK2WCNUOHLXCLQAXTYZYRTUWVA6AYQPN5UPQDZ2CID56XMP4H6XQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.018771406 (1h26m33.799704796s) [noise=true , address=104.36.82.47:4517 , piece_id=Y32KIPCALDESHPCDZTC5SAILDRHZH427OPTNWSA2J5UJ3BLRXGRQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.018907007 (1h26m33.7998404s) [noise=true , address=64.49.13.139:4478 , piece_id=PPZFKLQNUM2SUQX2MLYPFQLDVZ72WANLKJSD5STWOI7DFD7SOOIA, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.019355074 (1h26m33.800288457s) [noise=true , address=64.49.13.138:4483 , piece_id=LV72Z4XG6PDACBVTNRURVEDOHF6DTP6LUD6MF3K2IJ5CE7GY7M2Q, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.019718524 (1h26m33.800651907s) [noise=true , address=104.36.82.52:4099 , piece_id=U2JHRJRLRSFE5JN25LTDJFWCRI4CGL2BJ7CFVTPXZ5DQFLLNN7QQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.020086610 (1h26m33.801019997s) [noise=true , address=77.90.31.19:4860 , piece_id=2DABZ3ERA6BSOSKLDCZRDJJWFE5KULV4GR5EBFVO72O247DGPX3A, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.020448480 (1h26m33.801381863s) [noise=true , address=45.91.116.13:4175 , piece_id=LXPOY3TJAGAGIU2PJSRSXOG6YSAT36HFEBNLJUQHS3557TYNDFHQ, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.020778809 (1h26m33.801712195s) [noise=true , address=104.36.82.10:4096 , piece_id=7K4DJTT2SWSEDTT3ZOM5A4KM7OAOS4MFYLYM65X4AME22M2RCB5A, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.021340286 (1h26m33.802273684s) [noise=true , address=79.127.219.38:4004 , piece_id=RONGZNUD64KQITKQR6QVAWMWRIH4LJFBBXXCBBJD5RLROFBNZ7VA, seg_pos={0 0}, upload=1336]: Uploading piece... Jul 26 23:18:18.134472270 (1h26m33.915405653s) [upload=1336]: Next returned nil segment Jul 26 23:18:18.134498115 (1h26m33.915431502s) [upload=1336]: Waiting for error group managing segments... Jul 26 23:18:18.161612774 (1h26m33.94254615s) [noise=true , address=45.91.116.11:4151 , piece_id=37N7LUHOAW4PHDVEFRNIEQX4376CF3WEVVXINEE7CZKYLAMMN4RQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.161709861 (1h26m33.94264323s) [noise=true , address=104.36.82.54:4159 , piece_id=A3OBET2XDW7WKMLL4XCVQZ5WBVE633MY3U72LRCIWLFISREGPTZQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.192644699 (1h26m33.973578081s) [noise=true , address=77.90.31.21:4902 , piece_id=WEJPL4MORPE3WIQC44QD4KB2H3YU7TWW2MYBPBLX4Y2IZBAZI27Q, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.309825368 (1h26m34.090758733s) [noise=true , address=79.127.219.38:4004 , piece_id=RONGZNUD64KQITKQR6QVAWMWRIH4LJFBBXXCBBJD5RLROFBNZ7VA, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.635791122 (1h26m34.416724508s) [noise=true , address=79.127.205.233:4653 , piece_id=SB7L2LSSQKE3XTW7IKQQYJI4KRFDFRVQZV7LXEOCKZQCKGUSU6OQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.704468976 (1h26m34.485402345s) [noise=true , address=79.127.205.245:4225 , piece_id=MDPVVKF3JJJEJJRAP645ZBVV2LKDDIGOF2L4WJBNRTAMLT7NAJEA, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:18.809846319 (1h26m34.590779707s) [noise=true , address=77.90.31.13:4638 , piece_id=CEBNAK2FPBS44DMBK4MVYYBTYNIYCJKBR6L3X3R2QXJPWQ7SWKMA, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:23.139703097 (1h26m38.920636463s) [noise=true , address=64.49.13.138:4483 , piece_id=LV72Z4XG6PDACBVTNRURVEDOHF6DTP6LUD6MF3K2IJ5CE7GY7M2Q, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:23.220133506 (1h26m39.001066873s) [noise=true , address=45.91.116.14:4194 , piece_id=SK2WCNUOHLXCLQAXTYZYRTUWVA6AYQPN5UPQDZ2CID56XMP4H6XQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:24.564763574 (1h26m40.345696941s) [noise=true , address=64.49.13.139:4478 , piece_id=PPZFKLQNUM2SUQX2MLYPFQLDVZ72WANLKJSD5STWOI7DFD7SOOIA, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:28.650436015 (1h26m44.431369394s) [noise=true , address=45.91.116.13:4175 , piece_id=LXPOY3TJAGAGIU2PJSRSXOG6YSAT36HFEBNLJUQHS3557TYNDFHQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:33.004606556 (1h26m48.785539941s) [noise=true , address=77.90.31.19:4860 , piece_id=2DABZ3ERA6BSOSKLDCZRDJJWFE5KULV4GR5EBFVO72O247DGPX3A, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:18:51.458187265 (1h27m7.239120627s) [noise=true , address=79.127.201.215:4357 , piece_id=UAG3CLZA44VTWYFB4B74CG7I4U6JQFIEDAJBWC4AVROWYN7Y7W4Q, seg_pos={0 0}, upload=1336]: Done uploading piece. err: <nil> Jul 26 23:19:17.942913236 (1h27m33.723846606s) [noise=true , address=104.36.82.47:4517 , piece_id=Y32KIPCALDESHPCDZTC5SAILDRHZH427OPTNWSA2J5UJ3BLRXGRQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: rpc: dial tcp 104.36.82.47:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:17.943006298 (1h27m33.723939669s) [noise=true , address=104.36.82.52:4099 , piece_id=U2JHRJRLRSFE5JN25LTDJFWCRI4CGL2BJ7CFVTPXZ5DQFLLNN7QQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: ecclient: upload failed (node:122dGFWNmsuheRmfEfNGWHftPpCcWv7HHXKymgJxsye6P7ZKhZT, address:104.36.82.52:4099): protocol: write tcp 159.223.143.39:49052->104.36.82.52:4099: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:17.943097246 (1h27m33.724030632s) [noise=true , address=104.36.82.53:4129 , piece_id=2J3SPS2TCSSKWIV22J2DWMF65LNFIU4FPKU7VMSV3RCADJWBYMSQ, seg_pos={0 0}, upload=1336]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: write tcp 159.223.143.39:35032->104.36.82.53:4129: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:17.943192577 (1h27m33.724125957s) [noise=true , address=104.36.82.10:4096 , piece_id=7K4DJTT2SWSEDTT3ZOM5A4KM7OAOS4MFYLYM65X4AME22M2RCB5A, seg_pos={0 0}, upload=1336]: Done uploading piece. err: ecclient: upload failed (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA, address:104.36.82.10:4096): protocol: write tcp 159.223.143.39:44950->104.36.82.10:4096: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:17.943242578 (1h27m33.724175958s) [noise=true , address=104.36.82.46:4506 , piece_id=RROKRSO5GHA6NKRC7ZAIV6K4BUSIQCCA6XMEDC26WITGRFI35ANA, seg_pos={0 0}, upload=1336]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: write tcp 159.223.143.39:49472->104.36.82.46:4506: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:19:17.943269366 (1h27m33.724202753s) [seg_pos={0 0}, upload=1336]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:19:17.970217453 (1h27m33.751150847s) [upload=1336]: Done uploading segments... Jul 26 23:19:17.970248140 (1h27m33.751181525s) [upload=1336]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.030053831s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............................................XXX.X....XX.X.X] duration: 30.001644783s FAIL (2025-07-26 23:19:00 UTC) orbiter warning: context deadline exceeded Duration: 30.001668119s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 17 out of 60 is failed (threshold=3) [.............X........................X......XXXXXXXXXXXXXXX] duration: 30.03266378s FAIL (2025-07-26 23:18:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.032701707s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.XX.................X.X..........XX...X...X..XX.X...X..X....] duration: 33.242191064s FAIL (2025-07-26 23:18:44 UTC) context deadline exceeded Duration: 33.242207357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 30.049722696s FAIL (2025-07-26 23:18:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 525296s) [noise=true , address=45.91.116.4:4938 , piece_id=63DPNTD3GFXJ7IY3OKRGMIC4WGZ4D3EVKP4V3KQFOKZOGDWBML3A, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.279545133 (1h26m16.060478523s) [noise=true , address=104.36.82.2:4729 , piece_id=G6QQVY57BUVXXXH7Y724JVTBLLRBCVAN7MACG23UZ2Z57FJVWFYQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.280789217 (1h26m16.061722607s) [noise=true , address=45.91.116.14:4191 , piece_id=X6RMIDVE5G46AU2MTXCXELVCKHDFJPPYUIMU5L62DFBFOC5W75AA, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.282241259 (1h26m16.06317464s) [noise=true , address=104.36.82.8:4971 , piece_id=FLVKBFJQNVZTW2B3WRDMGKVPNNNJEY7YTGKLENSDR6QJER2JE32A, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.292222168 (1h26m16.073155587s) [noise=true , address=104.36.82.4:4802 , piece_id=DS37Z4SGBS6GAPJHE6QQJRNYG2FLO5JVW556OXVRCSPGKDZU4IPQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.301089564 (1h26m16.082022953s) [noise=true , address=45.91.116.12:4226 , piece_id=XAUCDHZ54UH52REUL5YSJRP4EPAZAWUJMGA4N5RYJQYXAUV2BWVQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.305127147 (1h26m16.086060537s) [noise=true , address=104.36.82.2:4731 , piece_id=PQWR56TKNZFB6KC5UBKIG5RIJMTHHUZOFPHJC5AUEO7N4SRRPTVA, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.308462552 (1h26m16.089395938s) [noise=true , address=45.91.116.7:4011 , piece_id=67NFFWCGIZS4XE5FZ7G4VYLHOSULQNNR7QSSUZCPGRDWHBBLOYFQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.309207438 (1h26m16.090140828s) [noise=true , address=104.36.82.5:4838 , piece_id=ERVCAMGCRELCFK4N3Z7XDTI57O7YDIL2DYLJIZW6JE3CNVNY3IJQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.310172279 (1h26m16.09110567s) [noise=true , address=104.36.82.10:4077 , piece_id=CSLHKKTCR3E65ST27D4QNAIO5HAJFO2EBR6JGJHSGG2YWER2R3CQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.314176624 (1h26m16.095110011s) [noise=true , address=104.36.82.7:4910 , piece_id=CBHM774RCIRDR75WJEAA4WPZ7GKL4LDLGUUD6MIB5PAPPK5MIVSQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.321471285 (1h26m16.102404672s) [noise=true , address=104.36.82.13:4187 , piece_id=QUPS4ZIJC5HVFO2E3IVNLYXKVIR5HOYR2I2MNFZF7OO4DUGMQGFA, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.327408186 (1h26m16.108341565s) [noise=true , address=104.36.82.15:4259 , piece_id=EO6AL7XRVOEK4Q2UG4YRG66CTLF2J6LGQWT3BSEE4ZIBMQ3AQ4MQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.332643861 (1h26m16.11357725s) [noise=true , address=104.36.82.11:4115 , piece_id=W6QKDCJOUTT63R62I2PH7CFIF5REM4ABYHLBLDMOVMSWEDKINJLQ, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.351499096 (1h26m16.132432486s) [noise=true , address=77.90.31.10:4518 , piece_id=AZWHPHYYSW4DTYG2E4KY2FH5375VC7EXSQIZ5N6ZIN3D3FRTEUBA, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.355740739 (1h26m16.136674126s) [noise=true , address=104.36.82.7:4909 , piece_id=X2TICLLMHNWCH5ZD6NYOSXPPNHVJF5RFAOSUTDERWENJTSIONLBA, seg_pos={0 0}, upload=1330]: Uploading piece... Jul 26 23:18:00.435795809 (1h26m16.216729186s) [upload=1330]: Next returned nil segment Jul 26 23:18:00.435818234 (1h26m16.21675162s) [upload=1330]: Waiting for error group managing segments... Jul 26 23:18:00.441544843 (1h26m16.222478211s) [noise=true , address=45.91.116.7:4011 , piece_id=67NFFWCGIZS4XE5FZ7G4VYLHOSULQNNR7QSSUZCPGRDWHBBLOYFQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: <nil> Jul 26 23:18:00.441626722 (1h26m16.222560101s) [noise=true , address=45.91.116.12:4226 , piece_id=XAUCDHZ54UH52REUL5YSJRP4EPAZAWUJMGA4N5RYJQYXAUV2BWVQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: <nil> Jul 26 23:18:00.442183212 (1h26m16.223116578s) [noise=true , address=45.91.116.14:4191 , piece_id=X6RMIDVE5G46AU2MTXCXELVCKHDFJPPYUIMU5L62DFBFOC5W75AA, seg_pos={0 0}, upload=1330]: Done uploading piece. err: <nil> Jul 26 23:18:00.443858435 (1h26m16.224791831s) [noise=true , address=45.91.116.4:4938 , piece_id=63DPNTD3GFXJ7IY3OKRGMIC4WGZ4D3EVKP4V3KQFOKZOGDWBML3A, seg_pos={0 0}, upload=1330]: Done uploading piece. err: <nil> Jul 26 23:18:07.722459075 (1h26m23.503392453s) [noise=true , address=104.36.82.13:4187 , piece_id=QUPS4ZIJC5HVFO2E3IVNLYXKVIR5HOYR2I2MNFZF7OO4DUGMQGFA, seg_pos={0 0}, upload=1330]: Done uploading piece. err: <nil> Jul 26 23:18:30.211825345 (1h26m45.992758724s) [noise=true , address=104.36.82.10:4077 , piece_id=CSLHKKTCR3E65ST27D4QNAIO5HAJFO2EBR6JGJHSGG2YWER2R3CQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:43120->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212493127 (1h26m45.993426492s) [noise=true , address=104.36.82.2:4729 , piece_id=G6QQVY57BUVXXXH7Y724JVTBLLRBCVAN7MACG23UZ2Z57FJVWFYQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212574327 (1h26m45.993507808s) [noise=true , address=104.36.82.7:4910 , piece_id=CBHM774RCIRDR75WJEAA4WPZ7GKL4LDLGUUD6MIB5PAPPK5MIVSQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 159.223.143.39:50510->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212653579 (1h26m45.99358696s) [noise=true , address=77.90.31.10:4518 , piece_id=AZWHPHYYSW4DTYG2E4KY2FH5375VC7EXSQIZ5N6ZIN3D3FRTEUBA, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 159.223.143.39:34986->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212740536 (1h26m45.993673914s) [noise=true , address=104.36.82.7:4909 , piece_id=X2TICLLMHNWCH5ZD6NYOSXPPNHVJF5RFAOSUTDERWENJTSIONLBA, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 159.223.143.39:58768->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212816157 (1h26m45.993749525s) [noise=true , address=104.36.82.8:4971 , piece_id=FLVKBFJQNVZTW2B3WRDMGKVPNNNJEY7YTGKLENSDR6QJER2JE32A, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:43036->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212884145 (1h26m45.993817516s) [noise=true , address=104.36.82.4:4802 , piece_id=DS37Z4SGBS6GAPJHE6QQJRNYG2FLO5JVW556OXVRCSPGKDZU4IPQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.212946298 (1h26m45.993879676s) [noise=true , address=104.36.82.15:4259 , piece_id=EO6AL7XRVOEK4Q2UG4YRG66CTLF2J6LGQWT3BSEE4ZIBMQ3AQ4MQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.213025924 (1h26m45.993959292s) [noise=true , address=104.36.82.11:4115 , piece_id=W6QKDCJOUTT63R62I2PH7CFIF5REM4ABYHLBLDMOVMSWEDKINJLQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:37348->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.213077537 (1h26m45.994010905s) [noise=true , address=104.36.82.2:4731 , piece_id=PQWR56TKNZFB6KC5UBKIG5RIJMTHHUZOFPHJC5AUEO7N4SRRPTVA, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 159.223.143.39:49002->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.213148364 (1h26m45.99408173s) [noise=true , address=104.36.82.5:4838 , piece_id=ERVCAMGCRELCFK4N3Z7XDTI57O7YDIL2DYLJIZW6JE3CNVNY3IJQ, seg_pos={0 0}, upload=1330]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:55074->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:30.213171019 (1h26m45.994104404s) [seg_pos={0 0}, upload=1330]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:18:30.247278023 (1h26m46.028211403s) [upload=1330]: Done uploading segments... Jul 26 23:18:30.247309019 (1h26m46.028242397s) [upload=1330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.049746186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXX] duration: 1m0.051942089s FAIL (2025-07-26 23:18:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== (1h25m33.679276158s) [seg_pos={0 0}, upload=1321]: Begin upload segment... Jul 26 23:17:17.898487866 (1h25m33.67942128s) [noise=true , address=77.90.31.18:4837 , piece_id=SR4GCYSEDIH3CRH7KNWQIKIXKOZK2NRTC4MZLKYXFHZO7ESO25VA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.899136989 (1h25m33.680070386s) [seg_pos={0 0}, upload=1321]: Done begin upload segment. Jul 26 23:17:17.899207466 (1h25m33.680140851s) [noise=true , address=64.49.13.170:4721 , piece_id=WWK27EQYJYGWBTOHF3B6ZXAOMJQ2HKNUOXYXGXPKTX62BEEOBYCQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.899447357 (1h25m33.680380753s) [noise=true , address=79.127.205.235:4688 , piece_id=FY4PM3YJ5AQROIMCOBJ2P2DU4AHZ2SPMQTVOQTFNASV6PC4NZQVA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.899650562 (1h25m33.680583929s) [noise=true , address=77.90.31.15:4703 , piece_id=4TOASMN3ABGPOZCAKYADVEVTXCXKCCJI76H7EPQ3ZFU7NJRU66AA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.900182475 (1h25m33.681115864s) [noise=true , address=45.91.116.5:4307 , piece_id=FBQOGXKFKZG6RYX45CEJJDUYBKY7USRFSM5GST42ODASUSMVXXAA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.900432091 (1h25m33.681365521s) [noise=true , address=104.36.82.56:4255 , piece_id=XM4JONWFX7UYNOIZFVMBP5DXAOJCUSFYVKAUECFZQTUJBDONGVZA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.900750299 (1h25m33.681683686s) [noise=true , address=104.36.82.47:4542 , piece_id=JADKC4MFZAOPUPL3BRXDDOLUKIEXCN2F5MQKVAB2MFTZT6GFFOFQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.900825191 (1h25m33.681758568s) [noise=true , address=104.36.82.46:4492 , piece_id=L7CWCXATU6F4NWE6DKN6XGQYUX5732IIRTBLLCQISHPQMTSTVIOA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.901129414 (1h25m33.682062791s) [noise=true , address=45.91.116.11:4151 , piece_id=FFL5B7C2WYTVCZNJHAZCBYJJB7NBXVHZLA7YRXRHNNFCGQUX4I2Q, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.901233930 (1h25m33.682167317s) [noise=true , address=77.90.31.11:4575 , piece_id=V7OFPGJJIJS364AV4DU5PHDEUIXQFSWSJOADV6BY4NPDF255HKLQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.901432735 (1h25m33.682366111s) [noise=true , address=64.49.13.140:4508 , piece_id=7CSAZQHUZEG33LD3B5NGKNGCJZ37UUBQMP7F276D5GEJDDUFMENA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.901722834 (1h25m33.682656211s) [noise=true , address=104.36.82.53:4138 , piece_id=USPNABGWEQ7BR2KSGPHBIECOTITHZK2SNULGSFYYSWNGPM2SGZUA, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.903061321 (1h25m33.683994716s) [noise=true , address=104.36.82.14:4235 , piece_id=BCLYC7SJUICCD32S7GOAAGZH7O7JOC7AVX3QRKW5FZLXANXRX6BQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.903803835 (1h25m33.684737226s) [noise=true , address=156.146.43.225:4434 , piece_id=3WFGSWUFBLHIQE6QAYBPCL7EESTJEWSVKZBQXAVLF3FBDBTE7HAQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.904644102 (1h25m33.68557749s) [noise=true , address=64.49.13.138:4482 , piece_id=CTYW2QA4KK3Q6IOB3IVW26WE7PSAUQQTVEGEC6R6IVNXGQLLRY2Q, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.905922425 (1h25m33.686855822s) [noise=true , address=77.90.31.16:4764 , piece_id=KWHZ4B5HIWIULJ7Q6AZIAYJ3MA56EGCNZNNIBGYIXKD46EPHPYYQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.907256071 (1h25m33.68818946s) [noise=true , address=45.91.116.14:4206 , piece_id=5F23262FGOUQTTTYPTL3SN3ZNUWKBBCJ76EK526OHPEIAIV4OFSQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.907843336 (1h25m33.688776726s) [noise=true , address=104.36.82.5:4868 , piece_id=FKKNGX3TZ3K2UFRRTW7XO2MAXFSSSP7UNSIE5ZJQDF2DX5TFGAAQ, seg_pos={0 0}, upload=1321]: Uploading piece... Jul 26 23:17:17.980250731 (1h25m33.761184104s) [upload=1321]: Next returned nil segment Jul 26 23:17:17.980272140 (1h25m33.761205518s) [upload=1321]: Waiting for error group managing segments... Jul 26 23:17:18.005533130 (1h25m33.78646651s) [noise=true , address=77.90.31.18:4837 , piece_id=SR4GCYSEDIH3CRH7KNWQIKIXKOZK2NRTC4MZLKYXFHZO7ESO25VA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:18.023420123 (1h25m33.8043535s) [noise=true , address=77.90.31.15:4703 , piece_id=4TOASMN3ABGPOZCAKYADVEVTXCXKCCJI76H7EPQ3ZFU7NJRU66AA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:18.076351467 (1h25m33.857284831s) [noise=true , address=77.90.31.11:4575 , piece_id=V7OFPGJJIJS364AV4DU5PHDEUIXQFSWSJOADV6BY4NPDF255HKLQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:18.088522435 (1h25m33.869455813s) [noise=true , address=77.90.31.16:4764 , piece_id=KWHZ4B5HIWIULJ7Q6AZIAYJ3MA56EGCNZNNIBGYIXKD46EPHPYYQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:18.482608960 (1h25m34.263542323s) [noise=true , address=79.127.205.235:4688 , piece_id=FY4PM3YJ5AQROIMCOBJ2P2DU4AHZ2SPMQTVOQTFNASV6PC4NZQVA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:19.494374274 (1h25m35.275307637s) [noise=true , address=45.91.116.5:4307 , piece_id=FBQOGXKFKZG6RYX45CEJJDUYBKY7USRFSM5GST42ODASUSMVXXAA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:19.749757663 (1h25m35.530691042s) [noise=true , address=45.91.116.11:4151 , piece_id=FFL5B7C2WYTVCZNJHAZCBYJJB7NBXVHZLA7YRXRHNNFCGQUX4I2Q, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: storage node overloaded, request limit: 2000 Jul 26 23:17:19.954329893 (1h25m35.735263274s) [noise=true , address=64.49.13.138:4482 , piece_id=CTYW2QA4KK3Q6IOB3IVW26WE7PSAUQQTVEGEC6R6IVNXGQLLRY2Q, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 23:17:21.016259136 (1h25m36.797192502s) [noise=true , address=156.146.43.225:4434 , piece_id=3WFGSWUFBLHIQE6QAYBPCL7EESTJEWSVKZBQXAVLF3FBDBTE7HAQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:21.697199166 (1h25m37.478132539s) [noise=true , address=64.49.13.170:4721 , piece_id=WWK27EQYJYGWBTOHF3B6ZXAOMJQ2HKNUOXYXGXPKTX62BEEOBYCQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:22.733238920 (1h25m38.51417229s) [noise=true , address=45.91.116.14:4206 , piece_id=5F23262FGOUQTTTYPTL3SN3ZNUWKBBCJ76EK526OHPEIAIV4OFSQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:17:24.554326344 (1h25m40.335259726s) [noise=true , address=64.49.13.140:4508 , piece_id=7CSAZQHUZEG33LD3B5NGKNGCJZ37UUBQMP7F276D5GEJDDUFMENA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: <nil> Jul 26 23:18:17.840751105 (1h26m33.621684499s) [noise=true , address=104.36.82.47:4542 , piece_id=JADKC4MFZAOPUPL3BRXDDOLUKIEXCN2F5MQKVAB2MFTZT6GFFOFQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: rpc: dial tcp 104.36.82.47:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.840904888 (1h26m33.621838269s) [noise=true , address=104.36.82.46:4492 , piece_id=L7CWCXATU6F4NWE6DKN6XGQYUX5732IIRTBLLCQISHPQMTSTVIOA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip, address:104.36.82.46:4492): protocol: rpc: dial tcp 104.36.82.46:4492: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.840979980 (1h26m33.621913359s) [noise=true , address=104.36.82.14:4235 , piece_id=BCLYC7SJUICCD32S7GOAAGZH7O7JOC7AVX3QRKW5FZLXANXRX6BQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV, address:104.36.82.14:4235): protocol: rpc: dial tcp 104.36.82.14:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.841056841 (1h26m33.621990227s) [noise=true , address=104.36.82.5:4868 , piece_id=FKKNGX3TZ3K2UFRRTW7XO2MAXFSSSP7UNSIE5ZJQDF2DX5TFGAAQ, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:18Srfics6dFkQeHZ2WF5YkFuLm2719Pw4Xgg9ntk12KnTBcn8v, address:104.36.82.5:4868): protocol: rpc: dial tcp 104.36.82.5:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.841115018 (1h26m33.622048399s) [noise=true , address=104.36.82.53:4138 , piece_id=USPNABGWEQ7BR2KSGPHBIECOTITHZK2SNULGSFYYSWNGPM2SGZUA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: rpc: dial tcp 104.36.82.53:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.841184398 (1h26m33.622117783s) [noise=true , address=104.36.82.56:4255 , piece_id=XM4JONWFX7UYNOIZFVMBP5DXAOJCUSFYVKAUECFZQTUJBDONGVZA, seg_pos={0 0}, upload=1321]: Done uploading piece. err: ecclient: upload failed (node:12S6rwimXBvaEb4KneAwL5H3t5zTmvGVhyJSWCTU1u17r4RWJta, address:104.36.82.56:4255): protocol: write tcp 159.223.143.39:42970->104.36.82.56:4255: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:18:17.841206883 (1h26m33.622140269s) [seg_pos={0 0}, upload=1321]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:18:17.872061485 (1h26m33.652994901s) [upload=1321]: Done uploading segments... Jul 26 23:18:17.872090760 (1h26m33.65302414s) [upload=1321]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.052055567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..XX.................X.X..........XX...X...X..XX.X...X..X...] duration: 35.837615673s FAIL (2025-07-26 23:18:11 UTC) context deadline exceeded Duration: 35.837638397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 16 out of 60 is failed (threshold=3) [..............X........................X......XXXXXXXXXXXXXX] duration: 30.035812192s FAIL (2025-07-26 23:18:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.035840949s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...XX.................X.X..........XX...X...X..XX.X...X..X..] duration: 52.078966636s FAIL (2025-07-26 23:17:35 UTC) context deadline exceeded Duration: 52.079279103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................................XXX.X....XX.X.] duration: 11.591963168s FAIL (2025-07-26 23:17:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.592019615s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [...............X........................X......XXXXXXXXXXXXX] duration: 30.042255229s FAIL (2025-07-26 23:17:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.042301243s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 30.065235507s FAIL (2025-07-26 23:17:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 58384044s) [upload=1311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:16:50.477460690 (1h25m6.258394074s) [upload=1311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:16:50.517769268 (1h25m6.298702652s) [seg_pos={0 0}, upload=1311]: Begin upload segment... Jul 26 23:16:50.517837176 (1h25m6.29877054s) [seg_pos={0 0}, upload=1311]: Done begin upload segment. Jul 26 23:16:50.517892715 (1h25m6.298826099s) [noise=true , address=45.91.116.5:4297 , piece_id=LYURRVXF2R77VFR3ZUCOV7FHIYJ6FVJBVCXZWWJ7I4XDS2WGJ6TQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.517904688 (1h25m6.298838077s) [noise=true , address=77.90.31.15:4698 , piece_id=YRE3VN2EOEUKY2FDBOLD4YPGFRFJIQE6V4ZJN5XOWK4DFZMIQP7Q, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.518417914 (1h25m6.299351302s) [noise=true , address=104.36.82.5:4840 , piece_id=SWINKVYHBBNUC2PURNJAKEYMLWYSBOA54FTSOHHCYN6V5IJYGA7Q, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.518532238 (1h25m6.299465625s) [noise=true , address=45.91.116.7:4011 , piece_id=N2AUHHVGEUKSIZA2R245GN5VRKDZK3FVQHTCOSONLILLLP6SRHWA, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.518929159 (1h25m6.299862542s) [noise=true , address=104.36.82.16:4293 , piece_id=HLG6C6RLI66UU3QKM4VXMQNUAZCKKULDWA6RCRET4KB73SYEMQFA, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.519425318 (1h25m6.300358702s) [noise=true , address=104.36.82.10:4079 , piece_id=GDB44HDEYZ244SCHSQLK3QMUDHBTDXCEODET7LPLBHRABE4JAZ3A, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.519843567 (1h25m6.300776951s) [noise=true , address=104.36.82.2:4732 , piece_id=PYP46NTVETKN5RQYGKPJHXXCUAKAL5RURDHTNGY3NPVKSDFVPMIA, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.521088839 (1h25m6.302022227s) [noise=true , address=104.36.82.6:4873 , piece_id=TTAITX3FCLOT7CJAVREZZQTMMVW2PTS43KZAHZAQ7TQJTYWS2MXQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.521274355 (1h25m6.302207743s) [noise=true , address=104.36.82.10:4078 , piece_id=LPDC6PQHHZ6AD7OM6H3Z6MRV5CDCF7RK6AOHJBX7AUG5VKBWLABQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.521728851 (1h25m6.302662241s) [noise=true , address=45.91.116.6:4975 , piece_id=4DOIAVU55WK46W6QSZPX2RHHSVAUBGQA5OVKWJJXCHBJYBXZ6L2Q, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.522203238 (1h25m6.303136641s) [noise=true , address=45.91.116.4:4940 , piece_id=WSMBJJGA6BGNV3KTN54W7T5MUFFID5XEAEWI3OSUSUPOA7UIAZYQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.522645516 (1h25m6.303578902s) [noise=true , address=104.36.82.15:4257 , piece_id=LH5O7SDUZ7LYHRAU55GZ73HX4KYLWHNH2KVVM6EVJZKBXMAPISFQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.523105751 (1h25m6.304039135s) [noise=true , address=104.36.82.9:4006 , piece_id=C3FWR7LAKHX77EY2MRCJHXHKRUDU2POJWD34MRYXSKC5JSV4L54Q, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.523528756 (1h25m6.304462143s) [noise=true , address=104.36.82.12:4151 , piece_id=DQEKMH7QGF5X2775A7UYL3IEJISGCB4OFKC72VJMCOZUXDH5I6QA, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.523959751 (1h25m6.304893138s) [noise=true , address=45.91.116.12:4227 , piece_id=TGYFKF6FDIDTHKXKFO35LYFRCIAPSRUG2WQKT6LKK2R6WAJGR2GQ, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.524394046 (1h25m6.305327418s) [noise=true , address=104.36.82.15:4259 , piece_id=V3OG6ROGD3ULBBJQLLKZXGGQU236P2RRX2MHGU3NYHA5YZUOWAAA, seg_pos={0 0}, upload=1311]: Uploading piece... Jul 26 23:16:50.624846792 (1h25m6.405780182s) [upload=1311]: Next returned nil segment Jul 26 23:16:50.624873984 (1h25m6.405807376s) [upload=1311]: Waiting for error group managing segments... Jul 26 23:16:50.629573272 (1h25m6.410506638s) [noise=true , address=45.91.116.7:4011 , piece_id=N2AUHHVGEUKSIZA2R245GN5VRKDZK3FVQHTCOSONLILLLP6SRHWA, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:16:50.630179183 (1h25m6.41111256s) [noise=true , address=45.91.116.6:4975 , piece_id=4DOIAVU55WK46W6QSZPX2RHHSVAUBGQA5OVKWJJXCHBJYBXZ6L2Q, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:16:50.630712922 (1h25m6.4116463s) [noise=true , address=45.91.116.4:4940 , piece_id=WSMBJJGA6BGNV3KTN54W7T5MUFFID5XEAEWI3OSUSUPOA7UIAZYQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:16:50.631271461 (1h25m6.412204842s) [noise=true , address=45.91.116.12:4227 , piece_id=TGYFKF6FDIDTHKXKFO35LYFRCIAPSRUG2WQKT6LKK2R6WAJGR2GQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:16:50.631865820 (1h25m6.412799191s) [noise=true , address=45.91.116.5:4297 , piece_id=LYURRVXF2R77VFR3ZUCOV7FHIYJ6FVJBVCXZWWJ7I4XDS2WGJ6TQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:16:50.648254688 (1h25m6.429188072s) [noise=true , address=77.90.31.15:4698 , piece_id=YRE3VN2EOEUKY2FDBOLD4YPGFRFJIQE6V4ZJN5XOWK4DFZMIQP7Q, seg_pos={0 0}, upload=1311]: Done uploading piece. err: <nil> Jul 26 23:17:20.439481673 (1h25m36.220415038s) [noise=true , address=104.36.82.2:4732 , piece_id=PYP46NTVETKN5RQYGKPJHXXCUAKAL5RURDHTNGY3NPVKSDFVPMIA, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.439631072 (1h25m36.220564434s) [noise=true , address=104.36.82.9:4006 , piece_id=C3FWR7LAKHX77EY2MRCJHXHKRUDU2POJWD34MRYXSKC5JSV4L54Q, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.439730850 (1h25m36.220664227s) [noise=true , address=104.36.82.10:4079 , piece_id=GDB44HDEYZ244SCHSQLK3QMUDHBTDXCEODET7LPLBHRABE4JAZ3A, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.439805025 (1h25m36.220738403s) [noise=true , address=104.36.82.15:4259 , piece_id=V3OG6ROGD3ULBBJQLLKZXGGQU236P2RRX2MHGU3NYHA5YZUOWAAA, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.439867887 (1h25m36.220801263s) [noise=true , address=104.36.82.6:4873 , piece_id=TTAITX3FCLOT7CJAVREZZQTMMVW2PTS43KZAHZAQ7TQJTYWS2MXQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:58148->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.440290437 (1h25m36.221223822s) [noise=true , address=104.36.82.10:4078 , piece_id=LPDC6PQHHZ6AD7OM6H3Z6MRV5CDCF7RK6AOHJBX7AUG5VKBWLABQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.440459329 (1h25m36.221392706s) [noise=true , address=104.36.82.15:4257 , piece_id=LH5O7SDUZ7LYHRAU55GZ73HX4KYLWHNH2KVVM6EVJZKBXMAPISFQ, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:57526->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.440588750 (1h25m36.221522127s) [noise=true , address=104.36.82.12:4151 , piece_id=DQEKMH7QGF5X2775A7UYL3IEJISGCB4OFKC72VJMCOZUXDH5I6QA, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.440697571 (1h25m36.221630948s) [noise=true , address=104.36.82.5:4840 , piece_id=SWINKVYHBBNUC2PURNJAKEYMLWYSBOA54FTSOHHCYN6V5IJYGA7Q, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.440815859 (1h25m36.221749236s) [noise=true , address=104.36.82.16:4293 , piece_id=HLG6C6RLI66UU3QKM4VXMQNUAZCKKULDWA6RCRET4KB73SYEMQFA, seg_pos={0 0}, upload=1311]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:48332->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:20.441167675 (1h25m36.222101065s) [seg_pos={0 0}, upload=1311]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:17:20.501459163 (1h25m36.282392549s) [upload=1311]: Done uploading segments... Jul 26 23:17:20.501484131 (1h25m36.282417522s) [upload=1311]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.065254163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXX] duration: 1m0.038425369s FAIL (2025-07-26 23:17:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== equestItem_SegmentBegin Jul 26 23:16:15.268298842 (1h24m31.049232227s) [seg_pos={0 0}, upload=1302]: Begin upload segment... Jul 26 23:16:15.268365966 (1h24m31.049299347s) [seg_pos={0 0}, upload=1302]: Done begin upload segment. Jul 26 23:16:15.268404721 (1h24m31.049338096s) [noise=true , address=104.36.82.56:4260 , piece_id=2PGOD55EA2ED3W56OQKYJ6UIV2G6XZTNDGTHZDM56XRXUUEBBBEA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.268412734 (1h24m31.049346115s) [noise=true , address=104.36.82.47:4545 , piece_id=L4XB777Z2VY5XJVMTJVXEAO5EK4T6CCJAHJ3Y6CQQRSA7X2H5YCA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.268893545 (1h24m31.049826923s) [noise=true , address=64.49.13.170:4721 , piece_id=JIAUBN26UC7BTKX7T4GUISAY74IENIABVFQKRRZNJNA7SHHWOOBA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.268926288 (1h24m31.049859668s) [noise=true , address=77.90.31.15:4726 , piece_id=MS4CIZJDWVCN27OEBEG6N4CEQ5TOW6456WDBMG6TUSHWCFUX3DVA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.269365569 (1h24m31.050298955s) [noise=true , address=104.36.82.52:4101 , piece_id=Y6HDBTTEI4HNFN7TB4ZEYOEVCARJ55MRFZHW4ARGVD2NO3XFJACA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.269419395 (1h24m31.05035277s) [noise=true , address=79.127.205.235:4682 , piece_id=5SS3RY4IEULQJFTRKBMLBEARQ67IFFEQVWKUL66T4CH3C44HMHFA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.269813259 (1h24m31.050746643s) [noise=true , address=79.127.205.245:4226 , piece_id=SG3ZOSR3LRKINCJFCMV56WLLKZL5HZH46UEQ2MFMMPXK6ZODCGFA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.269893057 (1h24m31.050826438s) [noise=true , address=104.36.82.11:4123 , piece_id=RCY7OFL3U5HF4UC22ODNKOLUTPAW2HL6IONKBYHMKD2F3TVXDHPA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.270239108 (1h24m31.051172492s) [noise=true , address=156.146.43.225:4449 , piece_id=ISQYFQRC57ANVPEBP5JXB6UMX7EAGDTIU25CFH523HRA4WMMES6Q, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.270392634 (1h24m31.05132602s) [noise=true , address=77.90.31.18:4821 , piece_id=C5VUWYWF2UNRPA6Z2RW7HTCSU3JL23RR7MVCHAGOW6AZHENAFP6A, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.270653190 (1h24m31.051586578s) [noise=true , address=79.127.201.215:4345 , piece_id=OA25U765MWY5IWCRLT4MYZ7MWQ4XGFRQXR6KHGLIKMTQ27PKIW5Q, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.270965440 (1h24m31.051898827s) [noise=true , address=79.127.205.234:4609 , piece_id=FUSNJM2K3NH6C557SNAJEWGGUAJSLYC7HO2CZA4UJPGI4AQEERPA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.271109225 (1h24m31.052042609s) [noise=true , address=104.36.82.55:4201 , piece_id=H7TDKFHQFUIIPR5KCXA664M5NEIKJJV432X2CBZQMS5YEJV7MWGQ, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.271547916 (1h24m31.052481339s) [noise=true , address=104.36.82.46:4495 , piece_id=2AFBZA3Z3L2LLMVHUCCCZZYQPMB7NNVAU7S7ZRQJ7O4WFNGCKNBQ, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.271579270 (1h24m31.052512647s) [noise=true , address=104.36.82.54:4173 , piece_id=AENJ36PKON2ZH23NUW2UZO3NYGWYQO4WSR7ZPZPZOKQY25XOHKEA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.272264283 (1h24m31.053197665s) [noise=true , address=77.90.31.11:4569 , piece_id=B77OVX2LDCZRWB23G7X26KYA25QB6M665O7GNB2SXSLJD2CUYHBQ, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.272699471 (1h24m31.053632856s) [noise=true , address=64.49.13.140:4508 , piece_id=LMSJM6GA35M2Y4YLGIAHEMQWU2MRXBYUKFNEOMFAKUXO7ACXQPOA, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.273635509 (1h24m31.054568878s) [noise=true , address=79.127.205.233:4647 , piece_id=KBD2YWTEXL64UFR6SM7J6JR7GTTO26UNJJRIYAMLJTBOJTSVAEHQ, seg_pos={0 0}, upload=1302]: Uploading piece... Jul 26 23:16:15.347248383 (1h24m31.128181765s) [upload=1302]: Next returned nil segment Jul 26 23:16:15.347271400 (1h24m31.128204778s) [upload=1302]: Waiting for error group managing segments... Jul 26 23:16:15.385700175 (1h24m31.166633555s) [noise=true , address=77.90.31.15:4726 , piece_id=MS4CIZJDWVCN27OEBEG6N4CEQ5TOW6456WDBMG6TUSHWCFUX3DVA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.440787186 (1h24m31.221720547s) [noise=true , address=77.90.31.11:4569 , piece_id=B77OVX2LDCZRWB23G7X26KYA25QB6M665O7GNB2SXSLJD2CUYHBQ, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.532146658 (1h24m31.313080036s) [noise=true , address=77.90.31.18:4821 , piece_id=C5VUWYWF2UNRPA6Z2RW7HTCSU3JL23RR7MVCHAGOW6AZHENAFP6A, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.862871119 (1h24m31.643804491s) [noise=true , address=79.127.205.245:4226 , piece_id=SG3ZOSR3LRKINCJFCMV56WLLKZL5HZH46UEQ2MFMMPXK6ZODCGFA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.868901356 (1h24m31.649834736s) [noise=true , address=79.127.205.234:4609 , piece_id=FUSNJM2K3NH6C557SNAJEWGGUAJSLYC7HO2CZA4UJPGI4AQEERPA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.879573398 (1h24m31.660506767s) [noise=true , address=79.127.205.233:4647 , piece_id=KBD2YWTEXL64UFR6SM7J6JR7GTTO26UNJJRIYAMLJTBOJTSVAEHQ, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:15.904954137 (1h24m31.685887506s) [noise=true , address=79.127.205.235:4682 , piece_id=5SS3RY4IEULQJFTRKBMLBEARQ67IFFEQVWKUL66T4CH3C44HMHFA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:21.848372982 (1h24m37.629306343s) [noise=true , address=64.49.13.140:4508 , piece_id=LMSJM6GA35M2Y4YLGIAHEMQWU2MRXBYUKFNEOMFAKUXO7ACXQPOA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:22.380953153 (1h24m38.161886517s) [noise=true , address=156.146.43.225:4449 , piece_id=ISQYFQRC57ANVPEBP5JXB6UMX7EAGDTIU25CFH523HRA4WMMES6Q, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:23.076539544 (1h24m38.85747292s) [noise=true , address=64.49.13.170:4721 , piece_id=JIAUBN26UC7BTKX7T4GUISAY74IENIABVFQKRRZNJNA7SHHWOOBA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:16:53.470850199 (1h25m9.251783563s) [noise=true , address=79.127.201.215:4345 , piece_id=OA25U765MWY5IWCRLT4MYZ7MWQ4XGFRQXR6KHGLIKMTQ27PKIW5Q, seg_pos={0 0}, upload=1302]: Done uploading piece. err: <nil> Jul 26 23:17:15.201468483 (1h25m30.982401861s) [noise=true , address=104.36.82.55:4201 , piece_id=H7TDKFHQFUIIPR5KCXA664M5NEIKJJV432X2CBZQMS5YEJV7MWGQ, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.201610806 (1h25m30.982544185s) [noise=true , address=104.36.82.46:4495 , piece_id=2AFBZA3Z3L2LLMVHUCCCZZYQPMB7NNVAU7S7ZRQJ7O4WFNGCKNBQ, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:1YpHarCJ3ouMGTs59udhU7az9h68ELEHTa2aTAax2chR2Gux1x, address:104.36.82.46:4495): protocol: rpc: dial tcp 104.36.82.46:4495: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.201720134 (1h25m30.982653508s) [noise=true , address=104.36.82.56:4260 , piece_id=2PGOD55EA2ED3W56OQKYJ6UIV2G6XZTNDGTHZDM56XRXUUEBBBEA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: rpc: dial tcp 104.36.82.56:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.201750754 (1h25m30.982684138s) [noise=true , address=104.36.82.47:4545 , piece_id=L4XB777Z2VY5XJVMTJVXEAO5EK4T6CCJAHJ3Y6CQQRSA7X2H5YCA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: write tcp 159.223.143.39:49546->104.36.82.47:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.201793937 (1h25m30.982727315s) [noise=true , address=104.36.82.11:4123 , piece_id=RCY7OFL3U5HF4UC22ODNKOLUTPAW2HL6IONKBYHMKD2F3TVXDHPA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:1LajLoxqMcmnKSCX6hVkthYVKZKTsPGphQHerwyjv4pi5GrXSE, address:104.36.82.11:4123): protocol: rpc: dial tcp 104.36.82.11:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.201949074 (1h25m30.982882458s) [noise=true , address=104.36.82.54:4173 , piece_id=AENJ36PKON2ZH23NUW2UZO3NYGWYQO4WSR7ZPZPZOKQY25XOHKEA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: rpc: dial tcp 104.36.82.54:4173: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.202335833 (1h25m30.983269213s) [noise=true , address=104.36.82.52:4101 , piece_id=Y6HDBTTEI4HNFN7TB4ZEYOEVCARJ55MRFZHW4ARGVD2NO3XFJACA, seg_pos={0 0}, upload=1302]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: write tcp 159.223.143.39:51004->104.36.82.52:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:17:15.202365325 (1h25m30.983298709s) [seg_pos={0 0}, upload=1302]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:17:15.234625329 (1h25m31.015558738s) [upload=1302]: Done uploading segments... Jul 26 23:17:15.234657510 (1h25m31.015590891s) [upload=1302]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.038468151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 30.026246927s FAIL (2025-07-26 23:16:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ding segments... Jul 26 23:16:20.280878777 (1h24m36.061812156s) [upload=1298]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:16:20.375333897 (1h24m36.156267258s) [upload=1305]: Starting upload Jul 26 23:16:20.375397763 (1h24m36.156331152s) [upload=1305]: Done starting upload Jul 26 23:16:20.375721527 (1h24m36.156654896s) [upload=1305]: Uploading segments... Jul 26 23:16:20.375787002 (1h24m36.156720381s) [upload=1305]: Got next segment. Inline: false Jul 26 23:16:20.375834274 (1h24m36.15676764s) [upload=1305]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:16:20.375839624 (1h24m36.15677299s) [upload=1305]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:16:20.403893360 (1h24m36.184826741s) [seg_pos={0 0}, upload=1305]: Begin upload segment... Jul 26 23:16:20.404128657 (1h24m36.18506202s) [seg_pos={0 0}, upload=1305]: Done begin upload segment. Jul 26 23:16:20.404164608 (1h24m36.185097989s) [noise=true , address=77.90.31.16:4735 , piece_id=EWKB67RZRW73IKGZPFGFHEVIXEKHXGPG56GBIYUD4LN75LLY2HGA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.404741840 (1h24m36.185675227s) [noise=true , address=77.90.31.14:4663 , piece_id=K4BHKWIO6CBXDLGCU5B6DOEEDBFDETNL24VNVE4HT6ME237HKW4Q, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.405206720 (1h24m36.186140108s) [noise=true , address=104.36.82.2:4730 , piece_id=JA32EI23QHJY3DS4TWRGHE56KE4NIKISA6ESYXFQ43TRH65EOJVQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.405691454 (1h24m36.186624841s) [noise=true , address=104.36.82.14:4223 , piece_id=ABARSEDKB6U445H6P65JDKII6YLQWKSWU6WM5HQZMF6EXRYTGLEA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.406149221 (1h24m36.187082609s) [noise=true , address=45.91.116.14:4190 , piece_id=IGG2UAWUPLGP7WPBUEP7WEPS7NK375QR34ZVGVQR5XHHDGHWXQPQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.406623605 (1h24m36.187557065s) [noise=true , address=45.91.116.4:4940 , piece_id=UB6UOLKRNMGH3X2OVYWNRIONTCZSGODCQ7ZDL6KEDAVW77JDJI6A, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.407406299 (1h24m36.188339687s) [noise=true , address=104.36.82.3:4765 , piece_id=FI4TTRKG4YNUWMZNFKQREBGUXLYYOZCLYAOUQEUVFMNIISSPIEZA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.408202208 (1h24m36.189135661s) [noise=true , address=104.36.82.7:4909 , piece_id=FNM3EIUZ3BF5RXOOU2PDTVDZGKOVBQJ3EPGHYIPBBV7HXUFIOMJQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.408995737 (1h24m36.189929122s) [noise=true , address=104.36.82.15:4259 , piece_id=3MB2GZ5THYJCLTMKEQ77BPQ4O37HJDKJNVVGTXEK2XE44HS6PPSA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.409487995 (1h24m36.190421428s) [noise=true , address=104.36.82.6:4873 , piece_id=MGRLY4IEVF7G2A2MKVPOWEBSGIF6GU6JIUBUGNFLZAWEPLLJHW2A, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.410122507 (1h24m36.191055895s) [noise=true , address=77.90.31.11:4553 , piece_id=MEZ74DFNRO5OJP6EIKK4H2YJHCNXAWBKVXQ6ZACOONVJK6PY43RA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.410662539 (1h24m36.191595925s) [noise=true , address=45.91.116.6:4975 , piece_id=KJTYAOJMGYKCC2WSWQVX2DODQ4KB3TAYPK6DTXXQQSPVWRSZPOGQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.411171041 (1h24m36.192104427s) [noise=true , address=45.91.116.6:4974 , piece_id=GOHPI3CUCV3YL432BTC6FL2TEQ2YKO4GU4IW6JVYN4MNTWKFPTRA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.411660403 (1h24m36.192593791s) [noise=true , address=45.91.116.11:4119 , piece_id=56NBVBYRPOTBTGOD4BEMZNSIUWKUZM566THTZBZGZP5JCC2I33FQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.412119588 (1h24m36.193052971s) [noise=true , address=104.36.82.4:4803 , piece_id=VHI36DWRX4UISAKQEGDEKYQNMPYDV5AIPCP2EDC2GYQMN4XTHVNA, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.412549647 (1h24m36.193483036s) [noise=true , address=45.91.116.5:4299 , piece_id=R7NFCL6DRY3RNGTBWBD2I3V5W7GO4U7CLWMA2MJ3ZTK5CSOHUEIQ, seg_pos={0 0}, upload=1305]: Uploading piece... Jul 26 23:16:20.499920520 (1h24m36.280853898s) [upload=1305]: Next returned nil segment Jul 26 23:16:20.499940464 (1h24m36.280873851s) [upload=1305]: Waiting for error group managing segments... Jul 26 23:16:20.505050235 (1h24m36.285983603s) [noise=true , address=45.91.116.5:4299 , piece_id=R7NFCL6DRY3RNGTBWBD2I3V5W7GO4U7CLWMA2MJ3ZTK5CSOHUEIQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.505127778 (1h24m36.286064938s) [noise=true , address=45.91.116.4:4940 , piece_id=UB6UOLKRNMGH3X2OVYWNRIONTCZSGODCQ7ZDL6KEDAVW77JDJI6A, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.505792186 (1h24m36.286725571s) [noise=true , address=77.90.31.11:4553 , piece_id=MEZ74DFNRO5OJP6EIKK4H2YJHCNXAWBKVXQ6ZACOONVJK6PY43RA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.505808539 (1h24m36.286741919s) [noise=true , address=45.91.116.14:4190 , piece_id=IGG2UAWUPLGP7WPBUEP7WEPS7NK375QR34ZVGVQR5XHHDGHWXQPQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.506377159 (1h24m36.28731054s) [noise=true , address=45.91.116.6:4975 , piece_id=KJTYAOJMGYKCC2WSWQVX2DODQ4KB3TAYPK6DTXXQQSPVWRSZPOGQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.506410063 (1h24m36.287343442s) [noise=true , address=45.91.116.6:4974 , piece_id=GOHPI3CUCV3YL432BTC6FL2TEQ2YKO4GU4IW6JVYN4MNTWKFPTRA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.509090453 (1h24m36.290023823s) [noise=true , address=77.90.31.14:4663 , piece_id=K4BHKWIO6CBXDLGCU5B6DOEEDBFDETNL24VNVE4HT6ME237HKW4Q, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:20.543011771 (1h24m36.323945133s) [noise=true , address=45.91.116.11:4119 , piece_id=56NBVBYRPOTBTGOD4BEMZNSIUWKUZM566THTZBZGZP5JCC2I33FQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:22.540735151 (1h24m38.321668513s) [noise=true , address=77.90.31.16:4735 , piece_id=EWKB67RZRW73IKGZPFGFHEVIXEKHXGPG56GBIYUD4LN75LLY2HGA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: <nil> Jul 26 23:16:50.344938979 (1h25m6.125872362s) [noise=true , address=104.36.82.15:4259 , piece_id=3MB2GZ5THYJCLTMKEQ77BPQ4O37HJDKJNVVGTXEK2XE44HS6PPSA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:39740->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345028052 (1h25m6.125961431s) [noise=true , address=104.36.82.7:4909 , piece_id=FNM3EIUZ3BF5RXOOU2PDTVDZGKOVBQJ3EPGHYIPBBV7HXUFIOMJQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345277383 (1h25m6.126210766s) [noise=true , address=104.36.82.2:4730 , piece_id=JA32EI23QHJY3DS4TWRGHE56KE4NIKISA6ESYXFQ43TRH65EOJVQ, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:47894->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345399979 (1h25m6.126333455s) [noise=true , address=104.36.82.6:4873 , piece_id=MGRLY4IEVF7G2A2MKVPOWEBSGIF6GU6JIUBUGNFLZAWEPLLJHW2A, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:33380->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345408872 (1h25m6.126342244s) [noise=true , address=104.36.82.14:4223 , piece_id=ABARSEDKB6U445H6P65JDKII6YLQWKSWU6WM5HQZMF6EXRYTGLEA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:36980->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345486772 (1h25m6.126420151s) [noise=true , address=104.36.82.4:4803 , piece_id=VHI36DWRX4UISAKQEGDEKYQNMPYDV5AIPCP2EDC2GYQMN4XTHVNA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 159.223.143.39:59728->104.36.82.4:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345500922 (1h25m6.126434285s) [noise=true , address=104.36.82.3:4765 , piece_id=FI4TTRKG4YNUWMZNFKQREBGUXLYYOZCLYAOUQEUVFMNIISSPIEZA, seg_pos={0 0}, upload=1305]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 159.223.143.39:57008->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:50.345572721 (1h25m6.126506105s) [seg_pos={0 0}, upload=1305]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:16:50.367347168 (1h25m6.14828055s) [upload=1305]: Done uploading segments... Jul 26 23:16:50.367399899 (1h25m6.148333283s) [upload=1305]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.026317707s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X...XX.................X.X..........XX...X...X..XX.X...X..X.] duration: 47.046638853s FAIL (2025-07-26 23:16:43 UTC) context deadline exceeded Duration: 47.046657492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................................XXX.X....XX.X] duration: 30.002753447s FAIL (2025-07-26 23:16:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002840455s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [................X........................X......XXXXXXXXXXXX] duration: 30.032847757s FAIL (2025-07-26 23:16:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.033667953s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 30.035113112s FAIL (2025-07-26 23:16:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== [upload=1298]: Got next segment. Inline: false Jul 26 23:15:50.279294479 (1h24m6.060227845s) [upload=1298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:15:50.279301109 (1h24m6.060234475s) [upload=1298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:15:50.307215353 (1h24m6.088148739s) [seg_pos={0 0}, upload=1298]: Begin upload segment... Jul 26 23:15:50.307301268 (1h24m6.088234654s) [seg_pos={0 0}, upload=1298]: Done begin upload segment. Jul 26 23:15:50.307314555 (1h24m6.08824794s) [noise=true , address=104.36.82.12:4150 , piece_id=4C6WEQD6BBT6ISOKRBIRU52N4NN53EXB7KZCUXDI4JSRMYFISD6Q, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.307331328 (1h24m6.08826471s) [noise=true , address=45.91.116.12:4226 , piece_id=3LTAMSRKVUIKI2QWYRXL7M2WN4LM6Y2ENN2HUW664VGMSL7APHQA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.307749007 (1h24m6.088682386s) [noise=true , address=77.90.31.14:4661 , piece_id=LBO6OIPOMGJJTDSBOCPSCYDUDJGQZ62LSRBQJWT5AZYRMPDJQVDA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.307863730 (1h24m6.088797115s) [noise=true , address=45.91.116.7:4011 , piece_id=7WTYEPTXOSTTOXAWU2NDYDHZNNVXOFCMTJ2PRVGODONOZKBBWIPQ, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.308461848 (1h24m6.089395223s) [noise=true , address=45.91.116.7:4010 , piece_id=SOI32TFCVJGLSJXDWY6LGWC5FK27BEI4RGHGWWJNKJQMN4TCR7SA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.308831042 (1h24m6.089764422s) [noise=true , address=104.36.82.10:4078 , piece_id=IFRROODHXTEYW5O4WL3EI4IQDBNV7565NNXTJKEJDURMDXZ3FH3A, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.308999953 (1h24m6.089933343s) [noise=true , address=104.36.82.10:4077 , piece_id=YSNWTGCPOXRUBZUYFSM3MZV32PMFBTGYF6RG6XNQ7TAMLJFDZNJA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.309480441 (1h24m6.090413825s) [noise=true , address=45.91.116.8:4048 , piece_id=AIH34MBLZPOAK456VHFENXVDA573CFOCKULIYMCYJXQALK3XGTBA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.309891269 (1h24m6.090824655s) [noise=true , address=104.36.82.11:4115 , piece_id=XF5LDPF742ZZHBR7UV5K3AHA6VZZOZSZ6C2S37TSKVXSMRDH4BEQ, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.310260739 (1h24m6.09119412s) [noise=true , address=45.91.116.6:4974 , piece_id=2IMF6TBQQQ77BXRQBC6TW3UMCJO7JXGL225UPVH4NMCILVVDHHPQ, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.310406206 (1h24m6.091339591s) [noise=true , address=104.36.82.14:4223 , piece_id=X5EE6YWLNWGFPXXHXPT7MNRMSOXXN5LHKN42TJDUQIAA3QWEG3FA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.310996588 (1h24m6.091929972s) [noise=true , address=45.91.116.13:4153 , piece_id=OHJT6EFXAZFYKDLUDGX4D2UCNCZY2OY5RFOJY3SXVBYATUHB5YLQ, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.311275061 (1h24m6.092208445s) [noise=true , address=104.36.82.7:4909 , piece_id=VW4677D73Q55RF43MXJD4AHHWVA5F5VJG5F3XPHOVZBMCTC452DQ, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.311702084 (1h24m6.092635473s) [noise=true , address=104.36.82.6:4874 , piece_id=QG4IR55URTAASLOZP37BUFO7UQ7FZY6IMPVE3VW7MGM34HR4GJQA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.312105707 (1h24m6.093039115s) [noise=true , address=104.36.82.15:4259 , piece_id=HW6MC6IDQPAARIBLOAE6PXA3LCKXJ35ZSG4ZL63DYS7NHQ7KYFXA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.312610980 (1h24m6.093544362s) [noise=true , address=104.36.82.2:4730 , piece_id=5R3K6OBC23JFHU6TKZETT75OV7TUXDJQ4QYRH33K6LHZI7WZ4LEA, seg_pos={0 0}, upload=1298]: Uploading piece... Jul 26 23:15:50.377543304 (1h24m6.158476672s) [upload=1298]: Next returned nil segment Jul 26 23:15:50.377558334 (1h24m6.158491709s) [upload=1298]: Waiting for error group managing segments... Jul 26 23:15:50.381622959 (1h24m6.162556328s) [noise=true , address=45.91.116.7:4011 , piece_id=7WTYEPTXOSTTOXAWU2NDYDHZNNVXOFCMTJ2PRVGODONOZKBBWIPQ, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.381725768 (1h24m6.162659143s) [noise=true , address=45.91.116.8:4048 , piece_id=AIH34MBLZPOAK456VHFENXVDA573CFOCKULIYMCYJXQALK3XGTBA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.382198192 (1h24m6.163131602s) [noise=true , address=45.91.116.6:4974 , piece_id=2IMF6TBQQQ77BXRQBC6TW3UMCJO7JXGL225UPVH4NMCILVVDHHPQ, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.382234818 (1h24m6.163168187s) [noise=true , address=45.91.116.7:4010 , piece_id=SOI32TFCVJGLSJXDWY6LGWC5FK27BEI4RGHGWWJNKJQMN4TCR7SA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.382606982 (1h24m6.163540347s) [noise=true , address=45.91.116.13:4153 , piece_id=OHJT6EFXAZFYKDLUDGX4D2UCNCZY2OY5RFOJY3SXVBYATUHB5YLQ, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.382656693 (1h24m6.163590057s) [noise=true , address=45.91.116.12:4226 , piece_id=3LTAMSRKVUIKI2QWYRXL7M2WN4LM6Y2ENN2HUW664VGMSL7APHQA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:15:50.383093682 (1h24m6.164027048s) [noise=true , address=77.90.31.14:4661 , piece_id=LBO6OIPOMGJJTDSBOCPSCYDUDJGQZ62LSRBQJWT5AZYRMPDJQVDA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: <nil> Jul 26 23:16:20.247621196 (1h24m36.028554572s) [noise=true , address=104.36.82.12:4150 , piece_id=4C6WEQD6BBT6ISOKRBIRU52N4NN53EXB7KZCUXDI4JSRMYFISD6Q, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:39762->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.247708578 (1h24m36.028641955s) [noise=true , address=104.36.82.6:4874 , piece_id=QG4IR55URTAASLOZP37BUFO7UQ7FZY6IMPVE3VW7MGM34HR4GJQA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:35944->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.247829726 (1h24m36.028763092s) [noise=true , address=104.36.82.2:4730 , piece_id=5R3K6OBC23JFHU6TKZETT75OV7TUXDJQ4QYRH33K6LHZI7WZ4LEA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:55314->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.247898812 (1h24m36.028832179s) [noise=true , address=104.36.82.11:4115 , piece_id=XF5LDPF742ZZHBR7UV5K3AHA6VZZOZSZ6C2S37TSKVXSMRDH4BEQ, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:38640->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.248293937 (1h24m36.029227312s) [noise=true , address=104.36.82.14:4223 , piece_id=X5EE6YWLNWGFPXXHXPT7MNRMSOXXN5LHKN42TJDUQIAA3QWEG3FA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:57882->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.248374212 (1h24m36.02930758s) [noise=true , address=104.36.82.15:4259 , piece_id=HW6MC6IDQPAARIBLOAE6PXA3LCKXJ35ZSG4ZL63DYS7NHQ7KYFXA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:47310->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.248391018 (1h24m36.029324382s) [noise=true , address=104.36.82.10:4078 , piece_id=IFRROODHXTEYW5O4WL3EI4IQDBNV7565NNXTJKEJDURMDXZ3FH3A, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:58836->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.248425810 (1h24m36.029359176s) [noise=true , address=104.36.82.10:4077 , piece_id=YSNWTGCPOXRUBZUYFSM3MZV32PMFBTGYF6RG6XNQ7TAMLJFDZNJA, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:35130->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:20.248538809 (1h24m36.029472185s) [noise=true , address=104.36.82.7:4909 , piece_id=VW4677D73Q55RF43MXJD4AHHWVA5F5VJG5F3XPHOVZBMCTC452DQ, seg_pos={0 0}, upload=1298]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:16:20.248610185 (1h24m36.02954356s) [seg_pos={0 0}, upload=1298]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:16:20.280846396 (1h24m36.06177976s) [upload=1298]: Done uploading segments... Jul 26 23:16:20.280878777 (1h24m36.061812156s) [upload=1298]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.035145579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXX] duration: 1m0.030273335s FAIL (2025-07-26 23:16:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== pload=1287]: Uploading piece... Jul 26 23:15:07.972509291 (1h23m23.75344268s) [noise=true , address=104.36.82.11:4121 , piece_id=HUHD7H3OQD2IK43YMUVDEFXFXZAFP77FRFRBEF6IGX2OBZFEFYHA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.973080267 (1h23m23.754013697s) [noise=true , address=104.36.82.53:4122 , piece_id=FEUI7GYJZLYQWLTPF2MGUQGHU64XUIDVR3CS6K5LOIBQ75A3URYA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.973562022 (1h23m23.754495409s) [noise=true , address=45.91.116.9:4273 , piece_id=JVNX4GTSC54GRKPZVFLLQHC2KDZRC47KBP6BXQDMAL37K5XEIGGA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.974036552 (1h23m23.754969951s) [noise=true , address=79.127.205.235:4686 , piece_id=FQFJS5F3BZKRYMMY7DE7OJSIVMDG4HPQJLNOXYKWSZAMMSWJMRFA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.974562455 (1h23m23.755495863s) [noise=true , address=104.36.82.46:4506 , piece_id=67NG5XPCAIIYY53GEKDCCYRRJTQ3LDPELSFY2PXJT54A2IJCOEOA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975053871 (1h23m23.755987255s) [noise=true , address=79.127.205.246:4284 , piece_id=G3O7BHQJSM5OL6VKT5QOAF4VENG7IMOUPWQ5THV7MZHRONBDMJCA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975145452 (1h23m23.756078836s) [noise=true , address=45.91.116.11:4121 , piece_id=72UBINQFQAAAB5DJNRNHSNXDUZTN75NLIGWO2XQLLEXQUZFREZEA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975520326 (1h23m23.756453715s) [noise=true , address=104.36.82.52:4090 , piece_id=PNBVLIRI47XNQAAKEFSHUJ72WUXGHNQWVEOYH3PRXVDOULSZOH3Q, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975537033 (1h23m23.75647041s) [noise=true , address=104.36.82.55:4206 , piece_id=Q6I5CAH7S6CISDOY3EWWLZTXDISMLKJ3KUQFW2OXAS5GGF2D43BQ, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975856702 (1h23m23.756790078s) [noise=true , address=77.90.31.15:4713 , piece_id=B6BKAO2PCS2APNPYT2PNRFOFBDJ362YMGFTJNZ66JSNTIAPF6ZNA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.975935292 (1h23m23.756868679s) [noise=true , address=64.49.13.140:4508 , piece_id=EXU4MEC7557C6G6F3LWBJC4PYVNH263E3AMCXMV6POQK22DRMTYQ, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.976330564 (1h23m23.757263944s) [noise=true , address=79.127.219.38:4009 , piece_id=SOTPCWYRPJGDVFUVIWGE33P2S67JV3YDT5P4OCGBLDE5TPH2YHUA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.976671258 (1h23m23.757604643s) [noise=true , address=104.36.82.40:4972 , piece_id=AGCNOB7SKHD44T6UYXGUMS7SNHXN4ZXT4FLX7JAHHZT4NJVWOCTQ, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.977024738 (1h23m23.757958118s) [noise=true , address=104.36.82.47:4529 , piece_id=QAOO3X4KVUGRKO5VF2LBPPD7GWRGIDZMVGZP4VOH4AFAOUYCYMPQ, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.977407411 (1h23m23.758340794s) [noise=true , address=156.146.43.225:4449 , piece_id=HJQLMMF3TYUDVZUB2B5LWPQIANUJH5WENRXPBPU3MEQMYUPWA45Q, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.977769767 (1h23m23.758703159s) [noise=true , address=104.36.82.5:4857 , piece_id=OEZDBT2CC2R5FSPRPMJBBI5WBJWRI4TPENARZ4LSFUHSRN4I26BQ, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:07.981218882 (1h23m23.762152252s) [noise=true , address=77.90.31.10:4537 , piece_id=AXDEUERUDLN5BHH5SUHPG63LZ2J6L4EAU4UYB53OFFPWWN3VAIIA, seg_pos={0 0}, upload=1287]: Uploading piece... Jul 26 23:15:08.049896429 (1h23m23.830829797s) [upload=1287]: Next returned nil segment Jul 26 23:15:08.049917676 (1h23m23.830851061s) [upload=1287]: Waiting for error group managing segments... Jul 26 23:15:08.085192460 (1h23m23.866125836s) [noise=true , address=79.127.219.38:4009 , piece_id=SOTPCWYRPJGDVFUVIWGE33P2S67JV3YDT5P4OCGBLDE5TPH2YHUA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:08.603450848 (1h23m24.384384234s) [noise=true , address=79.127.205.246:4284 , piece_id=G3O7BHQJSM5OL6VKT5QOAF4VENG7IMOUPWQ5THV7MZHRONBDMJCA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:08.665125001 (1h23m24.44605838s) [noise=true , address=79.127.205.235:4686 , piece_id=FQFJS5F3BZKRYMMY7DE7OJSIVMDG4HPQJLNOXYKWSZAMMSWJMRFA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:09.353269591 (1h23m25.134202959s) [noise=true , address=45.91.116.9:4273 , piece_id=JVNX4GTSC54GRKPZVFLLQHC2KDZRC47KBP6BXQDMAL37K5XEIGGA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:10.898271886 (1h23m26.679205253s) [noise=true , address=64.49.13.140:4508 , piece_id=EXU4MEC7557C6G6F3LWBJC4PYVNH263E3AMCXMV6POQK22DRMTYQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:10.931946288 (1h23m26.712879668s) [noise=true , address=77.90.31.15:4713 , piece_id=B6BKAO2PCS2APNPYT2PNRFOFBDJ362YMGFTJNZ66JSNTIAPF6ZNA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:12.100374758 (1h23m27.881308138s) [noise=true , address=156.146.43.225:4449 , piece_id=HJQLMMF3TYUDVZUB2B5LWPQIANUJH5WENRXPBPU3MEQMYUPWA45Q, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:14.060230501 (1h23m29.841163881s) [noise=true , address=45.91.116.11:4121 , piece_id=72UBINQFQAAAB5DJNRNHSNXDUZTN75NLIGWO2XQLLEXQUZFREZEA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:15:20.405371490 (1h23m36.18630487s) [noise=true , address=77.90.31.19:4860 , piece_id=WOUJICTIKWRWXIX4NWI3DQ52J7EBOZRG3YGFTKZBUR4PQGRU7KCQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: <nil> Jul 26 23:16:07.886099848 (1h24m23.667033231s) [noise=true , address=104.36.82.46:4506 , piece_id=67NG5XPCAIIYY53GEKDCCYRRJTQ3LDPELSFY2PXJT54A2IJCOEOA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: rpc: dial tcp 104.36.82.46:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886173501 (1h24m23.667106878s) [noise=true , address=104.36.82.53:4122 , piece_id=FEUI7GYJZLYQWLTPF2MGUQGHU64XUIDVR3CS6K5LOIBQ75A3URYA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: rpc: dial tcp 104.36.82.53:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886251860 (1h24m23.667185238s) [noise=true , address=104.36.82.55:4206 , piece_id=Q6I5CAH7S6CISDOY3EWWLZTXDISMLKJ3KUQFW2OXAS5GGF2D43BQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:12Mp4ZzYgrP3ZSXDeHbSTXaAJMPgGKUqVjdNtyfeKE5qBMAzBke, address:104.36.82.55:4206): protocol: rpc: dial tcp 104.36.82.55:4206: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886312734 (1h24m23.667246111s) [noise=true , address=104.36.82.40:4972 , piece_id=AGCNOB7SKHD44T6UYXGUMS7SNHXN4ZXT4FLX7JAHHZT4NJVWOCTQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:1XSuvCvq716GAa9WWmM3n2tqnfh8paaFUhjBzVzpiqwJtNifnr, address:104.36.82.40:4972): protocol: rpc: dial tcp 104.36.82.40:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886361447 (1h24m23.667294836s) [noise=true , address=104.36.82.52:4090 , piece_id=PNBVLIRI47XNQAAKEFSHUJ72WUXGHNQWVEOYH3PRXVDOULSZOH3Q, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: rpc: dial tcp 104.36.82.52:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886375590 (1h24m23.66730897s) [noise=true , address=104.36.82.5:4857 , piece_id=OEZDBT2CC2R5FSPRPMJBBI5WBJWRI4TPENARZ4LSFUHSRN4I26BQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: write tcp 159.223.143.39:51994->104.36.82.5:4857: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886423555 (1h24m23.667356934s) [noise=true , address=77.90.31.10:4537 , piece_id=AXDEUERUDLN5BHH5SUHPG63LZ2J6L4EAU4UYB53OFFPWWN3VAIIA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: rpc: dial tcp 77.90.31.10:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886459961 (1h24m23.667393328s) [noise=true , address=104.36.82.47:4529 , piece_id=QAOO3X4KVUGRKO5VF2LBPPD7GWRGIDZMVGZP4VOH4AFAOUYCYMPQ, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:1m7FrQEps7d8enRYk4v17UxcDs8o8L2CFcVEyB3zR4EniXrYeH, address:104.36.82.47:4529): protocol: rpc: dial tcp 104.36.82.47:4529: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886474039 (1h24m23.66740741s) [noise=true , address=104.36.82.11:4121 , piece_id=HUHD7H3OQD2IK43YMUVDEFXFXZAFP77FRFRBEF6IGX2OBZFEFYHA, seg_pos={0 0}, upload=1287]: Done uploading piece. err: ecclient: upload failed (node:1Fq1JNvrZLTFTQLVF8jzyavJ6dU1xX96FTGcyxfaHiPdCLu4pB, address:104.36.82.11:4121): protocol: rpc: dial tcp 104.36.82.11:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:16:07.886540011 (1h24m23.66747341s) [seg_pos={0 0}, upload=1287]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:16:07.913708503 (1h24m23.694641871s) [upload=1287]: Done uploading segments... Jul 26 23:16:07.913739321 (1h24m23.694672705s) [upload=1287]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.030293782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X...XX.................X.X..........XX...X...X..XX.X...X..X] duration: 1m0.000715343s FAIL (2025-07-26 23:15:56 UTC) context deadline exceeded Duration: 1m0.000740725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 30.035597826s FAIL (2025-07-26 23:15:49 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 0889088 (1h23m34.911822471s) [upload=1288]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:15:19.169160557 (1h23m34.950093969s) [seg_pos={0 0}, upload=1288]: Begin upload segment... Jul 26 23:15:19.169249223 (1h23m34.950182587s) [seg_pos={0 0}, upload=1288]: Done begin upload segment. Jul 26 23:15:19.169634165 (1h23m34.950567548s) [noise=true , address=77.90.31.14:4661 , piece_id=JI7JV5V7TECV64NNXDUX6T2RRTS7KBE37UYRYEXAK6LOGMWAYDFA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.169704244 (1h23m34.950637631s) [noise=true , address=45.91.116.11:4119 , piece_id=3IGAM3W7O5SXPU27QC673PQA22VI5HSEUMH6I4FZEZ2I5QW6LCIQ, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.170769336 (1h23m34.951702716s) [noise=true , address=104.36.82.15:4259 , piece_id=OZSLW6EVGNHMGXFWY33UYCW2DTMHJXOODQUOE6V55T4YA7DA2B3Q, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.171411757 (1h23m34.952345132s) [noise=true , address=45.91.116.12:4225 , piece_id=EIVGYLUO6DPC53OR3BOBB5SJEFURILWY5I57NTPMBKKDNPZQHTDA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.171553033 (1h23m34.952486425s) [noise=true , address=104.36.82.10:4077 , piece_id=YMSD2J5BSTZMEMTPWISIWXT5YONS3T5XZXDCGDFQCU4LX5I64SDA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.171803754 (1h23m34.952737147s) [noise=true , address=45.91.116.13:4154 , piece_id=YPMXNQ5QNAUGSUVCOA3TJM6CEW5FFVWDPY6QOTNC2GEXAPHDX75Q, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.172387449 (1h23m34.953320831s) [noise=true , address=104.36.82.6:4873 , piece_id=PYC5NUNYC3YUDWXK6IVCYXBX4DCGIRTYDK2IKP32T5DAZ2S4KTSQ, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.172405782 (1h23m34.953339168s) [noise=true , address=104.36.82.4:4804 , piece_id=I7ORPDGOPP3OO4RVG2UVK6ZY62CIHQFQFYUEOFOWBEKS2NF7QC5Q, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.172769369 (1h23m34.953702747s) [noise=true , address=45.91.116.4:4937 , piece_id=Q37QXZPFJALZJLK5LZFKRSL44HQOEMVCVDSWQMLCKBW7PAAJBPCA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.172827575 (1h23m34.953760958s) [noise=true , address=104.36.82.15:4258 , piece_id=7TQ34UE2ZQNQLGWKXF53JYFZUHIFJXFHSFAL5WSR6TWJS6D2TFZQ, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.173153487 (1h23m34.954086864s) [noise=true , address=104.36.82.12:4151 , piece_id=6NKLBOLCAWPZBFILJV7KA66GBYFMBV3B5SJYVVU5JFLUCEKU6OVA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.173195876 (1h23m34.954129264s) [noise=true , address=45.91.116.7:4009 , piece_id=YEHCEEUL74BXL3NGUWX4UCC7KADOR4YJUBWN6TFKKDMTGSMGR6MA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.173622931 (1h23m34.954556315s) [noise=true , address=104.36.82.8:4969 , piece_id=E52UAB65ZSA42EC27J2FRIM5CZUNQJFMM6A623QI7AFGEUZEBFZA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.174555913 (1h23m34.95548929s) [noise=true , address=104.36.82.14:4223 , piece_id=XZ4HBMPQNMK2REAS3Y5IIXUBZLR7V2AY3PEEPS442XS5QPNOLDVQ, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.174955719 (1h23m34.955889101s) [noise=true , address=104.36.82.9:4005 , piece_id=PJIBVQIQV6BYSBDL2I2LAUJQWYS7VSZ2AM4RMGRKP6UATINW6PWA, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.176640932 (1h23m34.957574314s) [noise=true , address=104.36.82.10:4078 , piece_id=2O25GYTYH7LYPGBVAZDDOKP5LQDCX4JWAZLYORUYWDROL4CUVUHQ, seg_pos={0 0}, upload=1288]: Uploading piece... Jul 26 23:15:19.267372139 (1h23m35.048305503s) [upload=1288]: Next returned nil segment Jul 26 23:15:19.267417139 (1h23m35.048350519s) [upload=1288]: Waiting for error group managing segments... Jul 26 23:15:19.272676888 (1h23m35.053610258s) [noise=true , address=45.91.116.4:4937 , piece_id=Q37QXZPFJALZJLK5LZFKRSL44HQOEMVCVDSWQMLCKBW7PAAJBPCA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:19.272742924 (1h23m35.053676295s) [noise=true , address=45.91.116.13:4154 , piece_id=YPMXNQ5QNAUGSUVCOA3TJM6CEW5FFVWDPY6QOTNC2GEXAPHDX75Q, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:19.273195386 (1h23m35.054128756s) [noise=true , address=45.91.116.7:4009 , piece_id=YEHCEEUL74BXL3NGUWX4UCC7KADOR4YJUBWN6TFKKDMTGSMGR6MA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:19.273254719 (1h23m35.054188095s) [noise=true , address=45.91.116.12:4225 , piece_id=EIVGYLUO6DPC53OR3BOBB5SJEFURILWY5I57NTPMBKKDNPZQHTDA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:19.273670316 (1h23m35.054603681s) [noise=true , address=77.90.31.14:4661 , piece_id=JI7JV5V7TECV64NNXDUX6T2RRTS7KBE37UYRYEXAK6LOGMWAYDFA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:19.284729689 (1h23m35.065663056s) [noise=true , address=45.91.116.11:4119 , piece_id=3IGAM3W7O5SXPU27QC673PQA22VI5HSEUMH6I4FZEZ2I5QW6LCIQ, seg_pos={0 0}, upload=1288]: Done uploading piece. err: <nil> Jul 26 23:15:49.089057876 (1h24m4.86999126s) [noise=true , address=104.36.82.14:4223 , piece_id=XZ4HBMPQNMK2REAS3Y5IIXUBZLR7V2AY3PEEPS442XS5QPNOLDVQ, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089145180 (1h24m4.870078556s) [noise=true , address=104.36.82.9:4005 , piece_id=PJIBVQIQV6BYSBDL2I2LAUJQWYS7VSZ2AM4RMGRKP6UATINW6PWA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089169179 (1h24m4.870102547s) [noise=true , address=104.36.82.10:4077 , piece_id=YMSD2J5BSTZMEMTPWISIWXT5YONS3T5XZXDCGDFQCU4LX5I64SDA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089231899 (1h24m4.870165288s) [noise=true , address=104.36.82.6:4873 , piece_id=PYC5NUNYC3YUDWXK6IVCYXBX4DCGIRTYDK2IKP32T5DAZ2S4KTSQ, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:59834->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089262394 (1h24m4.870195758s) [noise=true , address=104.36.82.15:4258 , piece_id=7TQ34UE2ZQNQLGWKXF53JYFZUHIFJXFHSFAL5WSR6TWJS6D2TFZQ, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089300387 (1h24m4.870233763s) [noise=true , address=104.36.82.4:4804 , piece_id=I7ORPDGOPP3OO4RVG2UVK6ZY62CIHQFQFYUEOFOWBEKS2NF7QC5Q, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089342821 (1h24m4.870276186s) [noise=true , address=104.36.82.8:4969 , piece_id=E52UAB65ZSA42EC27J2FRIM5CZUNQJFMM6A623QI7AFGEUZEBFZA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:57774->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089368407 (1h24m4.870301787s) [noise=true , address=104.36.82.15:4259 , piece_id=OZSLW6EVGNHMGXFWY33UYCW2DTMHJXOODQUOE6V55T4YA7DA2B3Q, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:56908->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089420664 (1h24m4.870354029s) [noise=true , address=104.36.82.12:4151 , piece_id=6NKLBOLCAWPZBFILJV7KA66GBYFMBV3B5SJYVVU5JFLUCEKU6OVA, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:42786->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089442128 (1h24m4.870375506s) [noise=true , address=104.36.82.10:4078 , piece_id=2O25GYTYH7LYPGBVAZDDOKP5LQDCX4JWAZLYORUYWDROL4CUVUHQ, seg_pos={0 0}, upload=1288]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:44946->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:49.089462712 (1h24m4.870396094s) [seg_pos={0 0}, upload=1288]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:15:49.121947666 (1h24m4.902881041s) [upload=1288]: Done uploading segments... Jul 26 23:15:49.121971551 (1h24m4.902904917s) [upload=1288]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 30.035650025s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................XXX.X....XX.] duration: 16.082839606s FAIL (2025-07-26 23:15:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.082859008s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [X................X........................X......XXXXXXXXXXX] duration: 30.036834536s FAIL (2025-07-26 23:15:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.036853296s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXX] duration: 1m0.031769906s FAIL (2025-07-26 23:15:07 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== address=104.36.82.54:4168 , piece_id=JS6DI5ATP522ZRWQXBJGVQJGEGNYTDAHFKVSQROHVBTAPIJGWJ2A, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.922621587 (1h22m23.703554999s) [noise=true , address=79.127.219.38:4018 , piece_id=ZZT3ZDSQVPL563RZHOESOXD3CNUOIZOTGZPX6U2C7VX3YMQVSLNA, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.922857781 (1h22m23.703791165s) [noise=true , address=77.90.31.18:4809 , piece_id=RZF7RMGOWWYPC3VE7QAOQT6JA7GL7GRZ2B7VOP3TAIEAI5ZEKPNA, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923220380 (1h22m23.70415377s) [noise=true , address=104.36.82.55:4202 , piece_id=HA6EOQICGIRALS4IQJK36GPEUNIYS7CFRG7VOTGZZUZSLC2S74CQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923273184 (1h22m23.704206563s) [noise=true , address=79.127.205.233:4650 , piece_id=33U5VOAFCDTXLGH6I72A23U5VOOJC556APTUUL2K3EUUEMEETISQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923577211 (1h22m23.704510592s) [noise=true , address=77.90.31.15:4714 , piece_id=O3T4MAGNMQKWZ5G4OGPJDMYNPF3YAW6SVBWPCWYL46VKI3A2NOQQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923623953 (1h22m23.704557331s) [noise=true , address=104.36.82.46:4481 , piece_id=67VLDMNQIXMLKQI67VOPISWFMKMQW7WKOYJFGGROSVQ5D3GWQBQQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923856441 (1h22m23.70478982s) [noise=true , address=104.36.82.53:4126 , piece_id=3R4O2LGB5K2LDMCVANNJVUYCGMCLPRENEQEMER5EB6RTURMFDP3A, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.923959847 (1h22m23.704893244s) [noise=true , address=104.36.82.47:4541 , piece_id=ZRKXL3BDWQJUKEMKI46W4URJDBCRA6FVTMTDNFE5VLTJLQISSCSQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.924400437 (1h22m23.705333821s) [noise=true , address=104.36.82.56:4249 , piece_id=KZLFAZW5WOGI72KZ4CWTULC2G2TL5UP442IOWITHPBCN2JG6WQOA, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.924549182 (1h22m23.705482563s) [noise=true , address=79.127.219.51:4212 , piece_id=BQA22URZHXDDFYVYTOLOLKXPHFTFQN7EKWARG4UHKY6ZDZGRKK7A, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.924756911 (1h22m23.70569032s) [noise=true , address=45.91.116.14:4206 , piece_id=3G7HFKGNSRME5H4SGHPECVFF2JRA2RPSA2DVE3KOT4F5XI245HNQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.924956049 (1h22m23.705889442s) [noise=true , address=64.49.13.140:4514 , piece_id=XX42UAGNHNQ6OB6S7FYCSAR6PQHNO43VWLCMMGHF6GMH4VN6WI6Q, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.925124183 (1h22m23.706057572s) [noise=true , address=77.90.31.10:4542 , piece_id=YQCGP3FQFOTBVMZTJIRQF3WDXJAZWCNPM6W45XW27XYFNVVPR6DA, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.925506588 (1h22m23.706439976s) [noise=true , address=64.49.13.155:4916 , piece_id=LKKPDPGWLR7XHBS7EYP5RUTDEW3F7XB226WBSRGK4B6LRL7BNA2Q, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.925516725 (1h22m23.7064501s) [noise=true , address=79.127.201.215:4341 , piece_id=CAJ6KHJDXGZ672I4JBR7GIKJONY3WAAMIOVJ7LKESUC33GOJ3OIA, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.926012823 (1h22m23.706946214s) [noise=true , address=104.36.82.39:4736 , piece_id=F7SMVZ2M6Q7LNGJNTEM6N7T4FDPHFY6IEGGGDLRBNMCWABFN2MEQ, seg_pos={0 0}, upload=1271]: Uploading piece... Jul 26 23:14:07.997679796 (1h22m23.77861318s) [upload=1271]: Next returned nil segment Jul 26 23:14:07.997702149 (1h22m23.778635539s) [upload=1271]: Waiting for error group managing segments... Jul 26 23:14:08.005146841 (1h22m23.786080204s) [noise=true , address=79.127.219.38:4018 , piece_id=ZZT3ZDSQVPL563RZHOESOXD3CNUOIZOTGZPX6U2C7VX3YMQVSLNA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:08.054275424 (1h22m23.835208794s) [noise=true , address=77.90.31.18:4809 , piece_id=RZF7RMGOWWYPC3VE7QAOQT6JA7GL7GRZ2B7VOP3TAIEAI5ZEKPNA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:08.185283145 (1h22m23.966216514s) [noise=true , address=79.127.201.215:4341 , piece_id=CAJ6KHJDXGZ672I4JBR7GIKJONY3WAAMIOVJ7LKESUC33GOJ3OIA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:08.474770242 (1h22m24.255703614s) [noise=true , address=45.91.116.14:4206 , piece_id=3G7HFKGNSRME5H4SGHPECVFF2JRA2RPSA2DVE3KOT4F5XI245HNQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:14:08.540594856 (1h22m24.321528227s) [noise=true , address=79.127.205.233:4650 , piece_id=33U5VOAFCDTXLGH6I72A23U5VOOJC556APTUUL2K3EUUEMEETISQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:08.545358834 (1h22m24.326292232s) [noise=true , address=79.127.205.235:4686 , piece_id=GAFVRN4QFOQW2LHMEHY4DS5RXTANVE72HZAS4YXBPCENWGWPXWVA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:09.429808929 (1h22m25.21074229s) [noise=true , address=64.49.13.155:4916 , piece_id=LKKPDPGWLR7XHBS7EYP5RUTDEW3F7XB226WBSRGK4B6LRL7BNA2Q, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:10.290889670 (1h22m26.071823045s) [noise=true , address=77.90.31.15:4714 , piece_id=O3T4MAGNMQKWZ5G4OGPJDMYNPF3YAW6SVBWPCWYL46VKI3A2NOQQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:10.983871167 (1h22m26.764804536s) [noise=true , address=64.49.13.140:4514 , piece_id=XX42UAGNHNQ6OB6S7FYCSAR6PQHNO43VWLCMMGHF6GMH4VN6WI6Q, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:14:11.096762579 (1h22m26.877695947s) [noise=true , address=79.127.219.51:4212 , piece_id=BQA22URZHXDDFYVYTOLOLKXPHFTFQN7EKWARG4UHKY6ZDZGRKK7A, seg_pos={0 0}, upload=1271]: Done uploading piece. err: <nil> Jul 26 23:15:07.791380082 (1h23m23.572313464s) [noise=true , address=104.36.82.46:4481 , piece_id=67VLDMNQIXMLKQI67VOPISWFMKMQW7WKOYJFGGROSVQ5D3GWQBQQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: rpc: dial tcp 104.36.82.46:4481: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791476152 (1h23m23.57240955s) [noise=true , address=104.36.82.53:4126 , piece_id=3R4O2LGB5K2LDMCVANNJVUYCGMCLPRENEQEMER5EB6RTURMFDP3A, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:104.36.82.53:4126): protocol: write tcp 159.223.143.39:35952->104.36.82.53:4126: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791553490 (1h23m23.572486869s) [noise=true , address=77.90.31.10:4542 , piece_id=YQCGP3FQFOTBVMZTJIRQF3WDXJAZWCNPM6W45XW27XYFNVVPR6DA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: write tcp 159.223.143.39:54566->77.90.31.10:4542: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791579072 (1h23m23.572512435s) [noise=true , address=104.36.82.56:4249 , piece_id=KZLFAZW5WOGI72KZ4CWTULC2G2TL5UP442IOWITHPBCN2JG6WQOA, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: write tcp 159.223.143.39:54678->104.36.82.56:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791646019 (1h23m23.572579381s) [noise=true , address=104.36.82.54:4168 , piece_id=JS6DI5ATP522ZRWQXBJGVQJGEGNYTDAHFKVSQROHVBTAPIJGWJ2A, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: write tcp 159.223.143.39:59834->104.36.82.54:4168: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791681935 (1h23m23.572615316s) [noise=true , address=104.36.82.39:4736 , piece_id=F7SMVZ2M6Q7LNGJNTEM6N7T4FDPHFY6IEGGGDLRBNMCWABFN2MEQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:126MWCnfveMZSncssS6wf4HGCi4eXvL7FvMD31oYoiR3SEFdGxX, address:104.36.82.39:4736): protocol: write tcp 159.223.143.39:33354->104.36.82.39:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791697549 (1h23m23.572630927s) [noise=true , address=104.36.82.55:4202 , piece_id=HA6EOQICGIRALS4IQJK36GPEUNIYS7CFRG7VOTGZZUZSLC2S74CQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: rpc: dial tcp 104.36.82.55:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791708847 (1h23m23.57264222s) [noise=true , address=104.36.82.47:4541 , piece_id=ZRKXL3BDWQJUKEMKI46W4URJDBCRA6FVTMTDNFE5VLTJLQISSCSQ, seg_pos={0 0}, upload=1271]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: rpc: dial tcp 104.36.82.47:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:15:07.791731505 (1h23m23.572664889s) [seg_pos={0 0}, upload=1271]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:15:07.819695565 (1h23m23.600628941s) [upload=1271]: Done uploading segments... Jul 26 23:15:07.819725538 (1h23m23.600658917s) [upload=1271]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.031794623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X................X........................X......XXXXXXXXXX] duration: 30.110176025s FAIL (2025-07-26 23:14:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.110203956s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................................XXX.X....XX] duration: 30.000612648s FAIL (2025-07-26 23:14:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000636428s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 30.027171395s FAIL (2025-07-26 23:14:36 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== g upload Jul 26 23:14:06.294674324 (1h22m22.075607705s) [upload=1270]: Done starting upload Jul 26 23:14:06.294951447 (1h22m22.075884817s) [upload=1270]: Uploading segments... Jul 26 23:14:06.295004832 (1h22m22.075938201s) [upload=1270]: Got next segment. Inline: false Jul 26 23:14:06.295097519 (1h22m22.076030898s) [upload=1270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:14:06.295103800 (1h22m22.076037177s) [upload=1270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:14:06.344934434 (1h22m22.125867808s) [seg_pos={0 0}, upload=1270]: Begin upload segment... Jul 26 23:14:06.344974552 (1h22m22.125907929s) [seg_pos={0 0}, upload=1270]: Done begin upload segment. Jul 26 23:14:06.344997847 (1h22m22.125931225s) [noise=true , address=104.36.82.11:4114 , piece_id=AFK76IEWKSPYQ426NQWPCLGYBAQ45P5SELOO4ASFCT6J6KEV32QQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.345583583 (1h22m22.12651697s) [noise=true , address=104.36.82.15:4259 , piece_id=I6DGUKCEOHCHLQHEKUU4CEQMSQ6KPKFZQKBNLQLSIQFO423IJIFA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.345790604 (1h22m22.12672399s) [noise=true , address=104.36.82.2:4730 , piece_id=ASDPUQNKNCQ753DOKKG6KC5CL4HBUBQ332GIZYHQZC3CUC5Z7SUA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.345967041 (1h22m22.126900429s) [noise=true , address=45.91.116.6:4974 , piece_id=ORGFBSGD424ROHJBGSY2UNPRNFQF6IMKWTE4QKDKUIEB6U4BHHAQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.346408223 (1h22m22.127341602s) [noise=true , address=77.90.31.14:4661 , piece_id=VR32LJN3C24CL6TZWB4SPF2QPGD73P65MXQNTANHA4E7UENNRO5A, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.346431140 (1h22m22.127364527s) [noise=true , address=45.91.116.5:4297 , piece_id=ATTQE6BLIF362ZKFLCBVJLWBHQD75GCOA2MIHLDG64MVOCZ7NOJQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.346844465 (1h22m22.127777843s) [noise=true , address=45.91.116.13:4155 , piece_id=XD7TPXEFNFGAI7MVQJXNIR3NG4HOCD7YAN4NMX6YSTNOG5VQEZEQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.346901249 (1h22m22.127834631s) [noise=true , address=104.36.82.8:4972 , piece_id=XXLEGIYETDZ2W6SU2AEWXIRN5VLVJW46KJBATRKSXBNJFVA6OEWA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.347365484 (1h22m22.128298868s) [noise=true , address=77.90.31.13:4626 , piece_id=U5VKQOPZKD62DL74PI7AN77OIKNJXA7FFQJ5RVAF73LBWDDOW7SA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.347460108 (1h22m22.128393489s) [noise=true , address=104.36.82.6:4873 , piece_id=MKFGJEZL5CB4SKBZLR5TV3VBHUKWIT7LLZY6I7PDUQJJV43SNJBQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.347779917 (1h22m22.128713292s) [noise=true , address=104.36.82.9:4006 , piece_id=QHZMOJPJZ265XXVY22SPXYBIDXCELFXH6S2Q3GLH3OOGE2MLZLPA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.348226301 (1h22m22.129159687s) [noise=true , address=45.91.116.8:4045 , piece_id=7XVKU2G2D44SDJGESGG3PJ3VG76RFBKOQGO5CEAPLXMGLZBHWZZA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.348707235 (1h22m22.129640613s) [noise=true , address=104.36.82.16:4295 , piece_id=ZCMU5GHJBSOUAAVTQXBXQSTARONRVQLMJDBJAIEAW6PQFKPARFLQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.349051019 (1h22m22.129984396s) [noise=true , address=45.91.116.7:4010 , piece_id=2TM2PQZEXY7L5C7W2ZSNSGJ6LAPTKJ4ECWTVPC3K6TOE4CUCGGLQ, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.349476364 (1h22m22.130409746s) [noise=true , address=104.36.82.4:4802 , piece_id=OVZAFKRPT354OEXFCTHZIGR2P5QY7GKZD4GNKXFAMDF4ZISFFK5A, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.349844580 (1h22m22.130777966s) [noise=true , address=104.36.82.14:4223 , piece_id=X35LXUFSX3GN355VTBJTIXJPAEG73YCVVSRWMNSROVYHDVFXSLHA, seg_pos={0 0}, upload=1270]: Uploading piece... Jul 26 23:14:06.444969814 (1h22m22.225903183s) [upload=1270]: Next returned nil segment Jul 26 23:14:06.444988096 (1h22m22.225921473s) [upload=1270]: Waiting for error group managing segments... Jul 26 23:14:06.451105122 (1h22m22.232038506s) [noise=true , address=77.90.31.13:4626 , piece_id=U5VKQOPZKD62DL74PI7AN77OIKNJXA7FFQJ5RVAF73LBWDDOW7SA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.451292045 (1h22m22.232225418s) [noise=true , address=45.91.116.5:4297 , piece_id=ATTQE6BLIF362ZKFLCBVJLWBHQD75GCOA2MIHLDG64MVOCZ7NOJQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.451795566 (1h22m22.232728937s) [noise=true , address=45.91.116.7:4010 , piece_id=2TM2PQZEXY7L5C7W2ZSNSGJ6LAPTKJ4ECWTVPC3K6TOE4CUCGGLQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.452269060 (1h22m22.233202437s) [noise=true , address=45.91.116.13:4155 , piece_id=XD7TPXEFNFGAI7MVQJXNIR3NG4HOCD7YAN4NMX6YSTNOG5VQEZEQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.452344082 (1h22m22.23327746s) [noise=true , address=45.91.116.8:4045 , piece_id=7XVKU2G2D44SDJGESGG3PJ3VG76RFBKOQGO5CEAPLXMGLZBHWZZA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.452541627 (1h22m22.233475006s) [noise=true , address=45.91.116.6:4974 , piece_id=ORGFBSGD424ROHJBGSY2UNPRNFQF6IMKWTE4QKDKUIEB6U4BHHAQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.458749432 (1h22m22.239682809s) [noise=true , address=77.90.31.14:4661 , piece_id=VR32LJN3C24CL6TZWB4SPF2QPGD73P65MXQNTANHA4E7UENNRO5A, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:06.473611380 (1h22m22.25454476s) [noise=true , address=104.36.82.8:4972 , piece_id=XXLEGIYETDZ2W6SU2AEWXIRN5VLVJW46KJBATRKSXBNJFVA6OEWA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: <nil> Jul 26 23:14:36.263512563 (1h22m52.044445961s) [noise=true , address=104.36.82.4:4802 , piece_id=OVZAFKRPT354OEXFCTHZIGR2P5QY7GKZD4GNKXFAMDF4ZISFFK5A, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.263626431 (1h22m52.044559808s) [noise=true , address=104.36.82.2:4730 , piece_id=ASDPUQNKNCQ753DOKKG6KC5CL4HBUBQ332GIZYHQZC3CUC5Z7SUA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.263694596 (1h22m52.044627974s) [noise=true , address=104.36.82.15:4259 , piece_id=I6DGUKCEOHCHLQHEKUU4CEQMSQ6KPKFZQKBNLQLSIQFO423IJIFA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:56564->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.263831405 (1h22m52.044764791s) [noise=true , address=104.36.82.9:4006 , piece_id=QHZMOJPJZ265XXVY22SPXYBIDXCELFXH6S2Q3GLH3OOGE2MLZLPA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:47842->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.263916310 (1h22m52.044849688s) [noise=true , address=104.36.82.14:4223 , piece_id=X35LXUFSX3GN355VTBJTIXJPAEG73YCVVSRWMNSROVYHDVFXSLHA, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:44766->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.264105934 (1h22m52.04503952s) [noise=true , address=104.36.82.11:4114 , piece_id=AFK76IEWKSPYQ426NQWPCLGYBAQ45P5SELOO4ASFCT6J6KEV32QQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:34796->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.264243249 (1h22m52.045176627s) [noise=true , address=104.36.82.6:4873 , piece_id=MKFGJEZL5CB4SKBZLR5TV3VBHUKWIT7LLZY6I7PDUQJJV43SNJBQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:36958->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.264252640 (1h22m52.045186019s) [noise=true , address=104.36.82.16:4295 , piece_id=ZCMU5GHJBSOUAAVTQXBXQSTARONRVQLMJDBJAIEAW6PQFKPARFLQ, seg_pos={0 0}, upload=1270]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:39936->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:14:36.264296199 (1h22m52.045229585s) [seg_pos={0 0}, upload=1270]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:14:36.287847086 (1h22m52.06878047s) [upload=1270]: Done uploading segments... Jul 26 23:14:36.287879918 (1h22m52.068813304s) [upload=1270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.027215315s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)