Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X..............X.......X....................................] duration: 1.290213662s FAIL (2025-06-23 02:20:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.290242642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.077929442s FAIL (2025-06-23 02:20:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.077964799s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.76095981s FAIL (2025-06-23 02:20:31 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.760990185s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X..............X.......X...................................] duration: 1.064655915s FAIL (2025-06-23 02:20:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.064680195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X..............X.......X..................................] duration: 1.595057885s FAIL (2025-06-23 02:20:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.595088826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.183595193s FAIL (2025-06-23 02:19:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.183619851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.779065676s FAIL (2025-06-23 02:19:18 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.779091501s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.338964833s FAIL (2025-06-23 02:19:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.338992146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X..............X.......X.................................] duration: 2.723410952s FAIL (2025-06-23 02:19:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.723451578s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.776650559s FAIL (2025-06-23 02:18:51 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.776703501s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.755872434s FAIL (2025-06-23 02:18:24 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.755918116s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X..............X.......X................................] duration: 1.025517277s FAIL (2025-06-23 02:18:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.025543947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 612.867665ms FAIL (2025-06-23 02:18:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 612.893721ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X..............X.......X...............................] duration: 6.055886404s FAIL (2025-06-23 02:17:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.055914702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 4.130360462s FAIL (2025-06-23 02:17:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.13039295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.764875389s FAIL (2025-06-23 02:17:24 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.764905599s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.398831146s FAIL (2025-06-23 02:17:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.398860293s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X..............X.......X..............................] duration: 4.983117887s FAIL (2025-06-23 02:16:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.983144908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X..............X.......X.............................] duration: 8.59531327s FAIL (2025-06-23 02:16:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.595345805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.799083539s FAIL (2025-06-23 02:16:10 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.799114382s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.625503067s FAIL (2025-06-23 02:16:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.625536553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.749023421s FAIL (2025-06-23 02:15:33 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.749062755s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X..............X.......X............................] duration: 739.79993ms FAIL (2025-06-23 02:15:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 739.850342ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.054628764s FAIL (2025-06-23 02:14:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.054661202s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.765894469s FAIL (2025-06-23 02:14:18 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.766003334s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X..............X.......X...........................] duration: 3.75889647s FAIL (2025-06-23 02:14:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.758921991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 3.649594922s FAIL (2025-06-23 02:13:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.649620925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.77038482s FAIL (2025-06-23 02:13:39 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.770410354s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X..............X.......X..........................] duration: 9.789385894s FAIL (2025-06-23 02:13:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.789414081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.690216745s FAIL (2025-06-23 02:12:48 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.690339423s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X...........................X...............................] duration: 3.970429568s FAIL (2025-06-23 02:12:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.970461456s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X..............X.......X.........................] duration: 7.453720241s FAIL (2025-06-23 02:12:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.453753682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X...........................X..............................] duration: 2.019604089s FAIL (2025-06-23 02:11:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.019630503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.923763991s FAIL (2025-06-23 02:11:38 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.923796807s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X..............X.......X........................] duration: 10.145527885s FAIL (2025-06-23 02:11:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.145549584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.758191462s FAIL (2025-06-23 02:11:17 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.75821838s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X...........................X.............................] duration: 2.663284437s FAIL (2025-06-23 02:11:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.663311667s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X..............X.......X.......................] duration: 652.951377ms FAIL (2025-06-23 02:10:28 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 652.973511ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X...........................X............................] duration: 3.985566775s FAIL (2025-06-23 02:10:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.985592803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.762921717s FAIL (2025-06-23 02:10:17 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.762962367s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X..............X.......X......................] duration: 2.783119391s FAIL (2025-06-23 02:09:21 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.783145685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.934858319s FAIL (2025-06-23 02:09:11 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.934883774s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X...........................X...........................] duration: 921.777198ms FAIL (2025-06-23 02:09:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 921.802184ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X...........................X..........................] duration: 3.089728771s FAIL (2025-06-23 02:08:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.089755006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.657691674s FAIL (2025-06-23 02:08:31 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.657724115s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X..............X.......X.....................] duration: 6.547748871s FAIL (2025-06-23 02:08:27 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.547780615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X..............X.......X....................] duration: 23.920416605s FAIL (2025-06-23 02:08:16 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.920433035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X...........................X.........................] duration: 25.984063358s FAIL (2025-06-23 02:07:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 25.984091727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.930539507s FAIL (2025-06-23 02:07:19 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.930568394s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X..............X.......X...................] duration: 5.855578222s FAIL (2025-06-23 02:07:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.855602531s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)