Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 967.411413ms FAIL (2025-06-23 02:39:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 967.438979ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.848941669s FAIL (2025-06-23 02:39:10 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.849043101s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.668120829s FAIL (2025-06-23 02:38:28 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.668155395s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.304199863s FAIL (2025-06-23 02:38:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.304276975s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.150128012s FAIL (2025-06-23 02:37:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.150160697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.870272718s FAIL (2025-06-23 02:37:28 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.870317076s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.92698394s FAIL (2025-06-23 02:36:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.927005327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.665466169s FAIL (2025-06-23 02:36:21 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.665499019s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.777613628s FAIL (2025-06-23 02:35:58 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.777704728s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 778.11636ms FAIL (2025-06-23 02:35:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 778.140566ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.663186554s FAIL (2025-06-23 02:34:43 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.663224876s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.513273386s FAIL (2025-06-23 02:34:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.513310713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.779307048s FAIL (2025-06-23 02:34:12 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.779334929s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 963.769148ms FAIL (2025-06-23 02:34:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 963.800845ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.820091948s FAIL (2025-06-23 02:34:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.820134152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.680916159s FAIL (2025-06-23 02:33:51 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.680950572s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.840521678s FAIL (2025-06-23 02:33:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.840604168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 498.788734ms FAIL (2025-06-23 02:33:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 498.818201ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X.......X...................................................] duration: 1.051485731s FAIL (2025-06-23 02:33:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.051525413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 693.298075ms FAIL (2025-06-23 02:33:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 693.340771ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.779020068s FAIL (2025-06-23 02:32:44 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.77910218s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X.......X..................................................] duration: 1.450304872s FAIL (2025-06-23 02:32:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.450327241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.514613393s FAIL (2025-06-23 02:32:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.514639627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.812050122s FAIL (2025-06-23 02:31:44 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.81208084s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X.......X.................................................] duration: 723.498417ms FAIL (2025-06-23 02:31:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 723.555875ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 704.692869ms FAIL (2025-06-23 02:31:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 704.720943ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.82515648s FAIL (2025-06-23 02:31:09 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.825186151s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.759180808s FAIL (2025-06-23 02:30:52 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.759210005s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X.......X................................................] duration: 714.511302ms FAIL (2025-06-23 02:30:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 714.536868ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 858.001846ms FAIL (2025-06-23 02:30:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 858.036429ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 570.799493ms FAIL (2025-06-23 02:30:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 570.834384ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.843288588s FAIL (2025-06-23 02:30:00 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.84332226s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X.......X...............................................] duration: 2.344662419s FAIL (2025-06-23 02:29:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.344694558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.743529113s FAIL (2025-06-23 02:29:00 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.743560074s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 584.901336ms FAIL (2025-06-23 02:28:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 584.929044ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.668728083s FAIL (2025-06-23 02:28:43 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.668759657s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 715.427656ms FAIL (2025-06-23 02:28:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 715.531913ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X.......X..............................................] duration: 1.637407812s FAIL (2025-06-23 02:28:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.637436129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.731162576s FAIL (2025-06-23 02:28:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.731220015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X.......X.............................................] duration: 2.25008488s FAIL (2025-06-23 02:28:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.250184564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.601819406s FAIL (2025-06-23 02:27:59 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.601844297s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.7033314s FAIL (2025-06-23 02:27:20 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.703363307s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 678.794808ms FAIL (2025-06-23 02:27:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 678.830842ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X.......X............................................] duration: 1.898111493s FAIL (2025-06-23 02:26:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.898145881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 631.914131ms FAIL (2025-06-23 02:26:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 631.963991ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X.......X...........................................] duration: 1.226492049s FAIL (2025-06-23 02:26:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.226516997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.7655008s FAIL (2025-06-23 02:26:13 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.765530346s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X.......X..........................................] duration: 874.138066ms FAIL (2025-06-23 02:26:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 874.160175ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 616.479426ms FAIL (2025-06-23 02:25:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 01:50:41.431554462 (105h27m16.994877451s) [upload=84382]: Uploading segments... Jun 23 01:50:41.431910238 (105h27m16.995233241s) [upload=84382]: Got next segment. Inline: false Jun 23 01:50:41.431998926 (105h27m16.995321916s) [upload=84382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:50:41.432004174 (105h27m16.995327173s) [upload=84382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:50:41.464462950 (105h27m17.027785939s) [seg_pos={0 0}, upload=84382]: Begin upload segment... Jun 23 01:50:41.464507487 (105h27m17.027830487s) [seg_pos={0 0}, upload=84382]: Done begin upload segment. Jun 23 01:50:41.464531075 (105h27m17.02785408s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.464921792 (105h27m17.028244811s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465158374 (105h27m17.028481365s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465373568 (105h27m17.028696567s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465506669 (105h27m17.028829672s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465731126 (105h27m17.029054128s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.465786215 (105h27m17.029109216s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466110238 (105h27m17.029433242s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466116708 (105h27m17.029439706s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466516127 (105h27m17.029839132s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466521840 (105h27m17.029844842s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466864149 (105h27m17.030187138s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.466934636 (105h27m17.030257646s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467233517 (105h27m17.03055652s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467353619 (105h27m17.030676623s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.467652593 (105h27m17.030975596s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.469282466 (105h27m17.032605475s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.471503462 (105h27m17.03482647s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Uploading piece... Jun 23 01:50:41.542320397 (105h27m17.105643384s) [upload=84382]: Next returned nil segment Jun 23 01:50:41.542343186 (105h27m17.105666194s) [upload=84382]: Waiting for error group managing segments... Jun 23 01:50:41.575152936 (105h27m17.138475936s) [noise=true , address=79.127.220.98:4410 , piece_id=ZB2XILUZ6QXGSQYIM5DSEXUBF423CU5HW2NM42PG6IJOR72LMVLQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500 Jun 23 01:50:41.615761887 (105h27m17.179084893s) [noise=true , address=77.90.31.14:4664 , piece_id=A3CO4P5P6R3OGYP672BVUMDHS54PAUF4WD3RMFOEBDO2Q26IKAHA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.799186218 (105h27m17.362509217s) [noise=true , address=79.127.205.245:4226 , piece_id=TNZ5RNHW2Q3SSFSQNNZCM7TSVHJPAAYH3OH2X2OLQLWJCLWK7YOQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12CeXmMFj7kBB8bmF2FsQs7HpNwu8o9Bx8y7MW1XqbK7LebdVfz, address:79.127.205.245:4226): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.819014262 (105h27m17.382337327s) [noise=true , address=45.91.116.8:5050 , piece_id=GSHW7BT3TNEJGJWSAFDTGLDKUNYP34OIPZ2QSMUHTXOCGTQIB76A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.852046895 (105h27m17.415369897s) [noise=true , address=79.127.205.234:5589 , piece_id=6OGXO6NK4TAI5SZYGGTAYGMEEEFEHVLNRJUP7D6YTC2HNF4JWGFA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12kKxHQJzDYNPj7h6TZTkuZYJBFgDEsn7RxPbLCPwN2AP1i6iSq, address:79.127.205.234:5589): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:41.947438848 (105h27m17.510761848s) [noise=true , address=45.91.116.9:5269 , piece_id=CNSKGZ2RLZL6L67QTBCNA2UOCVRLM5TRJDSNMDERNZJN55UGC7RA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:41.989891241 (105h27m17.553214252s) [noise=true , address=77.90.31.16:4746 , piece_id=N42MBEKCKB3MSCCA4EGGZLVF7B3EYLWYTMQMRNUVGO57UUB4G4YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.071240087 (105h27m17.634563087s) [noise=true , address=64.49.13.27:5968 , piece_id=TCRP6ITNLUGT4QFFGOOWFCAHC46CIVDVBHPPCRRDC3KOHDG4ZH7Q, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk, address:64.49.13.27:5968): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:42.265375612 (105h27m17.828698611s) [noise=true , address=104.36.82.52:4099 , piece_id=XSSJCLWBB6AL23E7PCV7IND5KG5JFRIITDJQYLRPUT4GC6YZSDUA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:43.084431024 (105h27m18.647754017s) [noise=true , address=104.36.82.16:6310 , piece_id=3FRCXY6MWF44EX3OFBMFP3KYOYFWUZ7HVEL432WDOTGKQ5SZ3SGQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:6310): protocol: storage node overloaded, request limit: 2000 Jun 23 01:50:44.242429086 (105h27m19.805752076s) [noise=true , address=79.127.219.51:4209 , piece_id=AROGZSPWWJ3CC5P57SJMTPT5DQKPEO7CPI3NMGEZER3PTDEH7YWQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:45.232388004 (105h27m20.795711007s) [noise=true , address=104.36.82.42:6814 , piece_id=Q4PFML7BVRAMLCUO4R4ILKLJJGDSMK5VZD3KLPI3M3AISEIUJ6GA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:50.182340950 (105h27m25.74566395s) [noise=true , address=77.90.31.21:4912 , piece_id=BZDXOVD2KRIOWMKOSK64HRTZF3NRTF57J7HHZZADF6YTXVDU2EEA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:50:57.925579034 (105h27m33.488902025s) [noise=true , address=104.36.82.56:4256 , piece_id=FI2SPTHCKW7YOADDHILXVKHOMNYKVD3PPE3W3I5DUN4W3UMN6PBQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:08.321586289 (105h27m43.884909274s) [noise=true , address=104.36.82.11:6132 , piece_id=D3WDLNJKSPUS6NXAHAOUKBWI6FHNP3EJVSXCRLWRXG67QXYFS3YQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:09.496771528 (105h27m45.060094678s) [noise=true , address=104.36.82.55:4210 , piece_id=7ONK2K3654VOC4KXDCEYY6FX6PWXI5KYIL5TSLEUDKTTRW3JEBTA, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:21.576931536 (105h27m57.140254527s) [noise=true , address=104.36.82.54:4184 , piece_id=ND547QP4R5IBLTB6UUJHAVICCPVHDBBUW2KXQGJW4FPZF7XQZNFQ, seg_pos={0 0}, upload=84382]: Done uploading piece. err: <nil> Jun 23 01:51:41.394109393 (105h28m16.957432393s) [noise=true , address=104.36.82.15:6291 , piece_id=6OQQCD32FZGRQIYULJPFS5QEXCUR6GNC65EMDLCAFICUAI4UFS7A, seg_pos={0 0}, upload=84382]: Done uploading piece. err: ecclient: upload failed (node:125JwnPDmKBNm3TybPWrzRbpzo5pnLei38joZeetvDG3wU32ZNb, address:104.36.82.15:6291): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 01:51:41.394152611 (105h28m16.957475623s) [seg_pos={0 0}, upload=84382]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jun 23 01:51:41.425935904 (105h28m16.989258895s) [upload=84382]: Done uploading segments... Jun 23 01:51:41.425962994 (105h28m16.989285997s) [upload=84382]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 616.511991ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.802888628s FAIL (2025-06-23 02:25:22 UTC) orbiter warning: uplink: bucket: metaclient: context deadline exceeded Duration: 2.802917862s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)