Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X...............X.] duration:
23.129434518s
FAIL (2025-06-22 22:59:08 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload
Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload
Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments...
Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false
Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment...
Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment.
Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000
Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment
Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments...
Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000
Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000
Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000
Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000
Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF
Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments...
Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 23.129458041s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X...............X] duration:
1m0.03429201s
FAIL (2025-06-22 22:58:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload
Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload
Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments...
Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false
Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment...
Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment.
Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece...
Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000
Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment
Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments...
Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000
Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000
Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000
Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000
Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil>
Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF
Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments...
Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.034319413s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................X................X....................] duration:
36.943223296s
FAIL (2025-06-22 22:58:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 36.94326644s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
23.213335488s
FAIL (2025-06-22 22:57:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 23.213369537s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................X................X...................] duration:
14.412499823s
FAIL (2025-06-22 22:57:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.412526723s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........................X................X..................] duration:
14.841594589s
FAIL (2025-06-22 22:57:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.841625924s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
25.59401267s
FAIL (2025-06-22 22:57:03 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 25.594040981s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
21.232062119s
FAIL (2025-06-22 22:56:11 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 21.232090934s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........................X................X.................] duration:
14.896381885s
FAIL (2025-06-22 22:56:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.896465334s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................X................X................] duration:
22.535152704s
FAIL (2025-06-22 22:55:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.53517851s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........................X................X...............] duration:
17.36429899s
FAIL (2025-06-22 22:55:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.365608613s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
19.045563257s
FAIL (2025-06-22 22:55:01 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 19.045590187s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............................X................X..............] duration:
28.956596911s
FAIL (2025-06-22 22:54:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.95661633s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
14.603465507s
FAIL (2025-06-22 22:54:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 14.603490688s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................X................X.............] duration:
41.437066367s
FAIL (2025-06-22 22:54:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 41.437164534s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
30.643265285s
FAIL (2025-06-22 22:53:16 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 30.643293329s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................X................X............] duration:
26.638517731s
FAIL (2025-06-22 22:52:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.638534655s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................X................X...........] duration:
19.059991055s
FAIL (2025-06-22 22:52:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.060109262s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
23.536097717s
FAIL (2025-06-22 22:52:27 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 23.536120313s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
22.813340311s
FAIL (2025-06-22 22:52:03 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 22.813362694s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
22.182211273s
FAIL (2025-06-22 22:51:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 22.182253351s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................X................X..........] duration:
27.594695126s
FAIL (2025-06-22 22:51:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 27.594731524s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
34.193290293s
FAIL (2025-06-22 22:50:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 34.193367822s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................X................X.........] duration:
34.366229037s
FAIL (2025-06-22 22:50:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 34.366267614s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
16.854152733s
FAIL (2025-06-22 22:50:04 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 16.854186664s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................X................X........] duration:
18.992551589s
FAIL (2025-06-22 22:49:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.992579568s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
14.665499396s
FAIL (2025-06-22 22:49:37 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 14.665537343s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................X................X.......] duration:
15.833235097s
FAIL (2025-06-22 22:49:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.833263811s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
30.875396771s
FAIL (2025-06-22 22:48:59 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 30.875424499s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................................X................X......] duration:
37.453744408s
FAIL (2025-06-22 22:48:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 37.453847209s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
14.630801192s
FAIL (2025-06-22 22:47:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 14.630825525s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X................X.....] duration:
42.598162346s
FAIL (2025-06-22 22:47:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 42.59819177s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
27.564075405s
FAIL (2025-06-22 22:47:18 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 27.564096642s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.02390793s
FAIL (2025-06-22 22:46:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.023924054s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................................X................X....] duration:
28.049737284s
FAIL (2025-06-22 22:46:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.04977082s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X................X...] duration:
19.557806092s
FAIL (2025-06-22 22:45:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.558670387s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
819.529561ms
FAIL (2025-06-22 22:45:04 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 819.560052ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........................................X................X..] duration:
25.059511477s
FAIL (2025-06-22 22:44:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 25.059532671s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
803.544658ms
FAIL (2025-06-22 22:44:07 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 803.576784ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X................X.] duration:
26.584225206s
FAIL (2025-06-22 22:43:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.584266216s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X................X] duration:
1m0.001480583s
FAIL (2025-06-22 22:43:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001501295s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
840.578297ms
FAIL (2025-06-22 22:42:59 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 840.608023ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
794.515333ms
FAIL (2025-06-22 22:42:37 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 794.611393ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
15.698568455s
FAIL (2025-06-22 22:41:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.698584861s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
800.060313ms
FAIL (2025-06-22 22:41:23 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 800.085359ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
57.027994971s
FAIL (2025-06-22 22:41:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 57.028019536s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
26.173965698s
FAIL (2025-06-22 22:40:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.173992561s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
827.578284ms
FAIL (2025-06-22 22:40:16 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 827.612386ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
26.156675961s
FAIL (2025-06-22 22:39:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.1567012s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
836.247046ms
FAIL (2025-06-22 22:39:06 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 836.273112ms
core
uplink
eu1
|