Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X...........................X........................] duration: 6.755605832s FAIL (2025-06-23 02:06:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.755629175s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X..............X.......X..................] duration: 9.391519143s FAIL (2025-06-23 02:06:54 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.391555787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.794512314s FAIL (2025-06-23 02:06:10 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.794542809s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................X..............X.......X.................] duration: 14.55528787s FAIL (2025-06-23 02:06:03 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.555332552s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X...........................X.......................] duration: 13.339127376s FAIL (2025-06-23 02:06:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 13.339160348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X...........................X......................] duration: 10.755132424s FAIL (2025-06-23 02:05:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 10.755171213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.70055317s FAIL (2025-06-23 02:05:29 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.700591045s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X...........................X.....................] duration: 13.024468359s FAIL (2025-06-23 02:05:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 13.024575721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X...................X..............X.......X................] duration: 5.150186013s FAIL (2025-06-23 02:05:11 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.150250694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.595633023s FAIL (2025-06-23 02:04:48 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.595680247s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X...........................X....................] duration: 16.551987707s FAIL (2025-06-23 02:04:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 16.552016415s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X...................X..............X.......X...............] duration: 7.531660392s FAIL (2025-06-23 02:04:03 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.53168843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.729593917s FAIL (2025-06-23 02:03:33 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.729625086s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X...........X...........................X...................] duration: 12.017886908s FAIL (2025-06-23 02:03:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 12.017916963s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X...................X..............X.......X..............] duration: 12.81745843s FAIL (2025-06-23 02:03:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.817481618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.671321259s FAIL (2025-06-23 02:03:09 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.671355495s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.748232407s FAIL (2025-06-23 02:02:47 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.748267074s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001223928s FAIL (2025-06-23 02:02:25 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 30.001247303s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X...........X...........................X..................] duration: 4.504595696s FAIL (2025-06-23 02:02:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.50461943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X...................X..............X.......X.............] duration: 5.471292883s FAIL (2025-06-23 02:02:07 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.47132099s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X...........X...........................X.................] duration: 34.833403846s FAIL (2025-06-23 02:02:05 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 34.833432392s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X...................X..............X.......X............] duration: 17.156256527s FAIL (2025-06-23 02:01:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.156281581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X...........X...........................X................] duration: 6.54515302s FAIL (2025-06-23 02:01:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.545179256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X...........X...........................X...............] duration: 1.069196658s FAIL (2025-06-23 02:00:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.069233907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X...................X..............X.......X...........] duration: 917.117236ms FAIL (2025-06-23 02:00:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 917.140452ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X...................X..............X.......X..........] duration: 48.936598198s FAIL (2025-06-23 02:00:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.936625644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X...........X...........................X..............] duration: 1.478863524s FAIL (2025-06-23 02:00:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.478983561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X...........X...........................X.............] duration: 8.272466342s FAIL (2025-06-23 02:00:02 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 8.272495545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X...........X...........................X............] duration: 8.767219346s FAIL (2025-06-23 01:59:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 8.767254064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X...........X...........................X...........] duration: 1.433568382s FAIL (2025-06-23 01:58:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.433598088s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X...................X..............X.......X.........] duration: 928.229117ms FAIL (2025-06-23 01:58:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 928.256183ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X...........X...........................X..........] duration: 4.18902515s FAIL (2025-06-23 01:58:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.189052689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X...................X..............X.......X........] duration: 7.962252386s FAIL (2025-06-23 01:58:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.96227818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X...........X...........................X.........] duration: 33.298119155s FAIL (2025-06-23 01:58:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 33.298147819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X...................X..............X.......X.......] duration: 16.982224209s FAIL (2025-06-23 01:57:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.982251833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X...........X...........................X........] duration: 1.529686199s FAIL (2025-06-23 01:57:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.5297078s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X...................X..............X.......X......] duration: 13.064109609s FAIL (2025-06-23 01:56:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.064136257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X...........X...........................X.......] duration: 26.556834949s FAIL (2025-06-23 01:56:22 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 26.55689489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X...........X...........................X......] duration: 9.099304583s FAIL (2025-06-23 01:55:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 9.099336051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X...................X..............X.......X.....] duration: 8.696974273s FAIL (2025-06-23 01:55:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.697020974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X...........X...........................X.....] duration: 34.739822139s FAIL (2025-06-23 01:54:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 34.739846657s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X...................X..............X.......X....] duration: 21.468190105s FAIL (2025-06-23 01:54:16 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.468213555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X...........X...........................X....] duration: 7.624310452s FAIL (2025-06-23 01:53:33 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 7.624334243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............X...................X..............X.......X...] duration: 8.003320421s FAIL (2025-06-23 01:53:32 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.00334576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X...........X...........................X...] duration: 25.097998752s FAIL (2025-06-23 01:53:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 25.098027757s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............X...................X..............X.......X..] duration: 27.963750254s FAIL (2025-06-23 01:53:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.963776709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X...........X...........................X..] duration: 17.082648735s FAIL (2025-06-23 01:52:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 17.082834514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............X...................X..............X.......X.] duration: 13.367476067s FAIL (2025-06-23 01:52:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.367494654s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X...........X...........................X.] duration: 23.995246213s FAIL (2025-06-23 01:52:13 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 23.995271033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................X...................X..............X.......X] duration: 1m0.001888528s FAIL (2025-06-23 01:52:04 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001920963s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)