Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X...................X..............X.......X............] duration: 17.156256527s FAIL (2025-06-23 02:01:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.156281581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X...........X...........................X................] duration: 6.54515302s FAIL (2025-06-23 02:01:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.545179256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X...........X...........................X...............] duration: 1.069196658s FAIL (2025-06-23 02:00:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.069233907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X...................X..............X.......X...........] duration: 917.117236ms FAIL (2025-06-23 02:00:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 917.140452ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X...................X..............X.......X..........] duration: 48.936598198s FAIL (2025-06-23 02:00:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.936625644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X...........X...........................X..............] duration: 1.478863524s FAIL (2025-06-23 02:00:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.478983561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X...........X...........................X.............] duration: 8.272466342s FAIL (2025-06-23 02:00:02 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 8.272495545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X...........X...........................X............] duration: 8.767219346s FAIL (2025-06-23 01:59:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 8.767254064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X...........X...........................X...........] duration: 1.433568382s FAIL (2025-06-23 01:58:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.433598088s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X...................X..............X.......X.........] duration: 928.229117ms FAIL (2025-06-23 01:58:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 928.256183ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X...........X...........................X..........] duration: 4.18902515s FAIL (2025-06-23 01:58:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.189052689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X...................X..............X.......X........] duration: 7.962252386s FAIL (2025-06-23 01:58:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.96227818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X...........X...........................X.........] duration: 33.298119155s FAIL (2025-06-23 01:58:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 33.298147819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X...................X..............X.......X.......] duration: 16.982224209s FAIL (2025-06-23 01:57:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.982251833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X...........X...........................X........] duration: 1.529686199s FAIL (2025-06-23 01:57:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.5297078s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X...................X..............X.......X......] duration: 13.064109609s FAIL (2025-06-23 01:56:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.064136257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X...........X...........................X.......] duration: 26.556834949s FAIL (2025-06-23 01:56:22 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 26.55689489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X...........X...........................X......] duration: 9.099304583s FAIL (2025-06-23 01:55:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 9.099336051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X...................X..............X.......X.....] duration: 8.696974273s FAIL (2025-06-23 01:55:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.697020974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X...........X...........................X.....] duration: 34.739822139s FAIL (2025-06-23 01:54:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 34.739846657s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X...................X..............X.......X....] duration: 21.468190105s FAIL (2025-06-23 01:54:16 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.468213555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X...........X...........................X....] duration: 7.624310452s FAIL (2025-06-23 01:53:33 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 7.624334243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X...................X..............X.......X...] duration: 8.003320421s FAIL (2025-06-23 01:53:32 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.00334576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X...........X...........................X...] duration: 25.097998752s FAIL (2025-06-23 01:53:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 25.098027757s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............X...................X..............X.......X..] duration: 27.963750254s FAIL (2025-06-23 01:53:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.963776709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X...........X...........................X..] duration: 17.082648735s FAIL (2025-06-23 01:52:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 17.082834514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............X...................X..............X.......X.] duration: 13.367476067s FAIL (2025-06-23 01:52:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.367494654s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X...........X...........................X.] duration: 23.995246213s FAIL (2025-06-23 01:52:13 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 23.995271033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................X...................X..............X.......X] duration: 1m0.001888528s FAIL (2025-06-23 01:52:04 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001920963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................X...........X...........................X] duration: 1m0.035133749s FAIL (2025-06-23 01:51:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.035161302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X...........X...........................] duration: 51.966435302s FAIL (2025-06-23 01:50:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 51.966484975s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X...................X..............X.......] duration: 10.010922349s FAIL (2025-06-23 01:50:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.010971547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X...................X..............X......] duration: 25.721787401s FAIL (2025-06-23 01:49:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.721811827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...........X..........................] duration: 22.065684356s FAIL (2025-06-23 01:49:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 22.065710826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................X...................X..............X.....] duration: 11.865419852s FAIL (2025-06-23 01:48:57 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.865446761s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X...........X.........................] duration: 20.873777676s FAIL (2025-06-23 01:48:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 20.873804526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...........X........................] duration: 37.211956345s FAIL (2025-06-23 01:47:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 37.211987808s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................X...................X..............X....] duration: 8.961910652s FAIL (2025-06-23 01:47:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.961932814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................X...................X..............X...] duration: 3.601896568s FAIL (2025-06-23 01:46:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.601959465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X...........X.......................] duration: 4.466693251s FAIL (2025-06-23 01:46:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 4.466725336s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X.....................X...................X..............X..] duration: 52.489394682s FAIL (2025-06-23 01:46:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.489414216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...........X......................] duration: 4.363261156s FAIL (2025-06-23 01:45:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 4.363293773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...........X.....................] duration: 19.179535248s FAIL (2025-06-23 01:45:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 19.179562425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...........X....................] duration: 32.313736783s FAIL (2025-06-23 01:45:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 32.313798351s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X.....................X...................X..............X.] duration: 1.238348389s FAIL (2025-06-23 01:44:58 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.238370797s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X.....................X...................X..............X] duration: 1m0.00051665s FAIL (2025-06-23 01:44:57 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000537883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X...........X...................] duration: 6.655932789s FAIL (2025-06-23 01:43:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 6.65596427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X.....................X...................X..............] duration: 15.986369356s FAIL (2025-06-23 01:43:01 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.986429686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...........X..................] duration: 14.583683993s FAIL (2025-06-23 01:42:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== oad=84082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 23 01:27:43.602110205 (105h4m19.165433214s) [upload=84082]: Done uploading segments... Jun 23 01:27:43.602149595 (105h4m19.165472598s) [upload=84082]: Upload finished. err: <nil> Jun 23 01:28:26.568078065 (105h5m2.131401068s) [upload=84097]: Starting upload Jun 23 01:28:26.568176623 (105h5m2.131499629s) [upload=84097]: Done starting upload Jun 23 01:28:26.568333698 (105h5m2.131656699s) [upload=84097]: Uploading segments... Jun 23 01:28:26.569025334 (105h5m2.132348341s) [upload=84097]: Got next segment. Inline: false Jun 23 01:28:26.569144232 (105h5m2.132467237s) [upload=84097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:28:26.569226194 (105h5m2.132549206s) [upload=84097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:28:26.603382637 (105h5m2.166705641s) [seg_pos={0 0}, upload=84097]: Begin upload segment... Jun 23 01:28:26.603439808 (105h5m2.16676282s) [seg_pos={0 0}, upload=84097]: Done begin upload segment. Jun 23 01:28:26.603481538 (105h5m2.166804559s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603498030 (105h5m2.166821033s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603958736 (105h5m2.16728174s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.603997417 (105h5m2.167320407s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604297025 (105h5m2.167620026s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.604692251 (105h5m2.16801525s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605044347 (105h5m2.168367349s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605369657 (105h5m2.168692667s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.605753081 (105h5m2.169076095s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606212150 (105h5m2.169535158s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606654203 (105h5m2.169977203s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.606961844 (105h5m2.170284849s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607374002 (105h5m2.170697008s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.607734796 (105h5m2.171057797s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608082032 (105h5m2.171405034s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.608428729 (105h5m2.171751737s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.610585268 (105h5m2.173908273s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.612001063 (105h5m2.175324064s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Uploading piece... Jun 23 01:28:26.687672089 (105h5m2.250995077s) [upload=84097]: Next returned nil segment Jun 23 01:28:26.687687969 (105h5m2.25101097s) [upload=84097]: Waiting for error group managing segments... Jun 23 01:28:26.707815366 (105h5m2.271138367s) [noise=true , address=79.127.220.98:4410 , piece_id=DLW3G47XXWAJ4FZAZDEL6TCIN5FKG26LUNVELDWSUE2GDH6IPBLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:28:26.735633260 (105h5m2.298956259s) [noise=true , address=104.36.82.21:6483 , piece_id=WTUXQB3XLSHBMZZQNK745AK5U3ZLLNKC3TLW4CEFA7OX4DJCOTVQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.793765419 (105h5m2.357088412s) [noise=true , address=156.146.43.225:4448 , piece_id=5SH6UKJRS56UOVGLLW7JKTSAQANRVHYVOMEXZ56UWYM3AEB7QGVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:26.868408971 (105h5m2.43173196s) [noise=true , address=77.90.31.15:4729 , piece_id=AAVBJ7VEQX67HQBXQ5DO3O5XJ37TRKU3PS2XXEH3WARMSEAQGQ2A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:26.989727871 (105h5m2.553050876s) [noise=true , address=45.91.116.8:5050 , piece_id=M4P3RKV2BUWYZOP2TOLNSMYOYDTOEG5NXXM6XZUYPLFQMLUGUIOA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.669519830 (105h5m3.23284283s) [noise=true , address=77.90.31.14:4693 , piece_id=YGA5IDHRAXIFCELCNCQF23PEFGJMFQOQHFWY3D7UODW5HYEIEWXA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:27.697322244 (105h5m3.260645244s) [noise=true , address=104.36.82.42:6830 , piece_id=KW3RNOZYENVRUR663PVSC4WFCLI3RG5YBITFXOYC6W2GTGEFXTVA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.071845632 (105h5m3.635168633s) [noise=true , address=104.36.82.55:4217 , piece_id=WQIZ3WHXBLCKNJ4ZVQRC2PQIRH6IVVO7J3UMM5V5GXEMODV36KLQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:28.589172169 (105h5m4.152495155s) [noise=true , address=45.91.116.9:5288 , piece_id=EQWQVX4JDLVTNRTKYNUIFGJYQUXPVUCUCYJPNVW2IYM3CGQY5D6A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:29.203146980 (105h5m4.76646998s) [noise=true , address=104.36.82.16:6310 , piece_id=CRHVWU4XYDGUG7XVGC3TI6DC2CLBTSTHND4YB2YFEFPBD6TOAXKQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:28:29.434402213 (105h5m4.997725203s) [noise=true , address=45.91.116.10:5084 , piece_id=WCIMJN6XDXUO2VCRKLAVOPVVQUEQDGHR4NP4KQGY27NEUPCD5OFA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.293775103 (105h5m5.857098094s) [noise=true , address=104.36.82.9:6021 , piece_id=4JQGGBGDZMQ6FDS35A7BZLB5OLZL2JHMD5Q3TXAETVX7TYA74EHQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.347258498 (105h5m5.910581502s) [noise=true , address=104.36.82.12:6174 , piece_id=VQG2SHTAARK4FOSQER2YRKHZV7ODG7ZDPHNF6AESZGE3GMITKZZA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:30.547306548 (105h5m6.110629586s) [noise=true , address=79.127.219.51:4209 , piece_id=ZHJC5QUBGOHT3HRY4XPPADV7CYINQRXT4ID53KVC6LYXM7IV533A, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:34.333414200 (105h5m9.896737197s) [noise=true , address=79.127.219.38:4031 , piece_id=TBIY2ZMYVVKMYV4LVGJEKCJXMXYVJKSZKKBXASVNEUU2P7DKZKIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:42.464111369 (105h5m18.02743436s) [noise=true , address=104.36.82.53:4142 , piece_id=CL2WQNZHZNZM5BFJQIBUBQ7FQCIJSK4AR7PR2OLBIXFULPMBTGIQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:28:49.581287518 (105h5m25.144610509s) [noise=true , address=104.36.82.3:5797 , piece_id=CJD4BHNHEVAGUJ5K55QM7OLAQQLEYRD4C6KJQBZ4FKJLDREGLNRQ, seg_pos={0 0}, upload=84097]: Done uploading piece. err: <nil> Jun 23 01:29:26.535692548 (105h6m2.099015547s) [noise=true , address=104.36.82.11:6131 , piece_id=BI4HK2NGEXXNEDYGRXRL77E5CONZG6JH4IWMONSB64DHOJLQQZPA, seg_pos={0 0}, upload=84097]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:6131): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:29:26.535746358 (105h6m2.099069362s) [seg_pos={0 0}, upload=84097]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:29:26.562546209 (105h6m2.125869225s) [upload=84097]: Done uploading segments... Jun 23 01:29:26.562593265 (105h6m2.125916275s) [upload=84097]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration:* 14.583724313s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X.....................X...................X.............] duration: 11.769142205s FAIL (2025-06-23 01:42:07 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.769161466s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)