Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X.......................................X.....] duration: 10.523085787s FAIL (2025-06-22 23:18:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.523351333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X...............X..........................] duration: 21.90532804s FAIL (2025-06-22 23:18:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 21.905347386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X...............X.........................] duration: 42.209049734s FAIL (2025-06-22 23:18:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 42.20907726s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X.......................................X....] duration: 30.265591093s FAIL (2025-06-22 23:18:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.265634135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X...............X........................] duration: 59.427196154s FAIL (2025-06-22 23:17:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 59.427222865s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X.......................................X...] duration: 33.29416255s FAIL (2025-06-22 23:17:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.294186982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X................X.......................................X..] duration: 18.446971405s FAIL (2025-06-22 23:16:49 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.447012903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X................X.......................................X.] duration: 49.111708442s FAIL (2025-06-22 23:16:30 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.112465548s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X...............X.......................] duration: 15.499386109s FAIL (2025-06-22 23:16:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 15.49943558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...............X......................] duration: 37.921090352s FAIL (2025-06-22 23:15:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 37.92112327s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X................X.......................................X] duration: 1m0.001413757s FAIL (2025-06-22 23:15:41 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001448681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X...............X.....................] duration: 28.262181719s FAIL (2025-06-22 23:15:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 28.262232157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X................X.......................................] duration: 12.760808323s FAIL (2025-06-22 23:14:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.760828816s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X................X......................................] duration: 28.328786962s FAIL (2025-06-22 23:14:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.328838371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...............X....................] duration: 21.28511401s FAIL (2025-06-22 23:14:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 21.285144904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X...............X...................] duration: 16.165139991s FAIL (2025-06-22 23:13:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 16.165160824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...............X..................] duration: 54.652012465s FAIL (2025-06-22 23:13:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 54.652048277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X................X.....................................] duration: 35.310124638s FAIL (2025-06-22 23:13:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.310141492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...............X.................] duration: 20.970356658s FAIL (2025-06-22 23:11:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 20.97039349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X................X....................................] duration: 13.236488276s FAIL (2025-06-22 23:11:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.236515993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...............X................] duration: 24.921131745s FAIL (2025-06-22 23:10:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 24.92117757s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X................X...................................] duration: 15.423511038s FAIL (2025-06-22 23:10:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.423537417s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X................X..................................] duration: 57.080198614s FAIL (2025-06-22 23:10:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 57.080217495s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X...............X...............] duration: 30.395135264s FAIL (2025-06-22 23:09:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 30.395156063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...............X..............] duration: 19.078365342s FAIL (2025-06-22 23:09:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.078395402s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X................X.................................] duration: 21.143319595s FAIL (2025-06-22 23:08:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.143345391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...............X.............] duration: 18.542578025s FAIL (2025-06-22 23:08:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 18.542612628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X................X................................] duration: 25.988200343s FAIL (2025-06-22 23:08:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.988229044s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X................X...............................] duration: 20.684392411s FAIL (2025-06-22 23:07:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.684421523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...............X............] duration: 26.110033569s FAIL (2025-06-22 23:07:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 26.110061092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X...............X...........] duration: 32.403102762s FAIL (2025-06-22 23:06:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 32.403125044s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X................X..............................] duration: 19.085051542s FAIL (2025-06-22 23:06:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.085092173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X................X.............................] duration: 27.684249794s FAIL (2025-06-22 23:06:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.684272078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...............X..........] duration: 27.548446937s FAIL (2025-06-22 23:06:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 27.548477068s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X................X............................] duration: 25.638319442s FAIL (2025-06-22 23:05:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.638344687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...............X.........] duration: 16.934446981s FAIL (2025-06-22 23:04:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 16.934474799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X................X...........................] duration: 21.03509941s FAIL (2025-06-22 23:04:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.035190713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...............X........] duration: 22.703429895s FAIL (2025-06-22 23:03:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 22.703465735s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X................X..........................] duration: 27.05784226s FAIL (2025-06-22 23:03:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.057863023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...............X.......] duration: 19.808333763s FAIL (2025-06-22 23:02:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 19.80837082s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X................X.........................] duration: 35.166570568s FAIL (2025-06-22 23:02:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.166603221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...............X......] duration: 20.755761693s FAIL (2025-06-22 23:02:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 20.755790715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...............X.....] duration: 16.545141108s FAIL (2025-06-22 23:01:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 16.545173098s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X................X........................] duration: 19.111473343s FAIL (2025-06-22 23:01:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.113113095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...............X....] duration: 21.931344164s FAIL (2025-06-22 23:00:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 21.931376557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X................X.......................] duration: 17.284772227s FAIL (2025-06-22 23:00:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.284803861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...............X...] duration: 18.291444438s FAIL (2025-06-22 23:00:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 18.291470337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...............X..] duration: 30.274445627s FAIL (2025-06-22 23:00:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:57:40.837928291 (102h34m16.401251281s) [upload=82052]: Starting upload Jun 22 22:57:40.838014658 (102h34m16.40133766s) [upload=82052]: Done starting upload Jun 22 22:57:40.838129679 (102h34m16.401452669s) [upload=82052]: Uploading segments... Jun 22 22:57:40.838295105 (102h34m16.401618094s) [upload=82052]: Got next segment. Inline: false Jun 22 22:57:40.838348818 (102h34m16.401671819s) [upload=82052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 22 22:57:40.838355016 (102h34m16.401678017s) [upload=82052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 22 22:57:40.872986557 (102h34m16.436309557s) [seg_pos={0 0}, upload=82052]: Begin upload segment... Jun 22 22:57:40.873054514 (102h34m16.436377504s) [seg_pos={0 0}, upload=82052]: Done begin upload segment. Jun 22 22:57:40.873110613 (102h34m16.436433615s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873549926 (102h34m16.436872936s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.873926324 (102h34m16.437249334s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874278924 (102h34m16.437601934s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874591497 (102h34m16.437914498s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.874888847 (102h34m16.438211855s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875137644 (102h34m16.438460677s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875262242 (102h34m16.438585247s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875526661 (102h34m16.43884965s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875560428 (102h34m16.438883434s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875854235 (102h34m16.43917724s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.875896223 (102h34m16.439219419s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876184259 (102h34m16.439507261s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876476858 (102h34m16.439799859s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.876758892 (102h34m16.440081897s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877186957 (102h34m16.440509966s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877515755 (102h34m16.440838758s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:40.877830532 (102h34m16.441153534s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Uploading piece... Jun 22 22:57:41.281013804 (102h34m16.844336804s) [noise=true , address=45.91.116.7:5019 , piece_id=SFMNWJ234NUZDONT34RF7IBBWXQJVZBD6G7D3ZL6H5GLH74Z3SJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:128Sx7BgHeTbxPF76pcRzJtHZKcWiYaboxcnkadhVLvs5AHYzFm, address:45.91.116.7:5019): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.321929293 (102h34m16.885252283s) [upload=82052]: Next returned nil segment Jun 22 22:57:41.322316828 (102h34m16.885639826s) [upload=82052]: Waiting for error group managing segments... Jun 22 22:57:41.383876079 (102h34m16.947199085s) [noise=true , address=156.146.43.225:4448 , piece_id=MVWNDNWTFQNV4A4PMTLCTJY5GWZMYUUVRUSNDRRYKKDHY4EC5Q6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.507808448 (102h34m17.071131449s) [noise=true , address=45.91.116.14:5206 , piece_id=O5VTXXB4CJE256QKKYWULWSDELWFAW5Z7HXFMLZ36OJD67BHOHEA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:5206): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:41.909140035 (102h34m17.472463025s) [noise=true , address=104.36.82.7:5924 , piece_id=GHOF7IU2XYEADEOTHKFGGDPDFQ7PDRE6E4MNRAJN465NCJ2DFI3Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:104.36.82.7:5924): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:42.735083659 (102h34m18.298406667s) [noise=true , address=77.90.31.14:4691 , piece_id=QUI3MSG5E3LG4RMKA4W2H6KNDSDUOP77LORAXACPB5MK7B52DRBQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:43.212222981 (102h34m18.775545989s) [noise=true , address=77.90.31.11:4578 , piece_id=6NQGGWU6WP62FGZH2CU3YNYF3NNSAGZ4FZ2HKWBT4QGKGVU3DK5A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: storage node overloaded, request limit: 2000 Jun 22 22:57:44.947587248 (102h34m20.510910238s) [noise=true , address=104.36.82.22:6511 , piece_id=P4XIRHQZX25QH72LWQL46VPSDWLXPQOHU2WFXZAARD726TEDMK6A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:45.681269493 (102h34m21.24459249s) [noise=true , address=104.36.82.10:6093 , piece_id=NHVSGPINVGEMHJYH227TWK3XGCFLLAWHBYJSJ32JXTPODTPHUQMA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:53.387807013 (102h34m28.951129999s) [noise=true , address=104.36.82.56:4256 , piece_id=Q5XJIMFENAHE7Z5PZINC4VBY227GBUQNWLWZDBCYYZMGXPC4GCPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:57:57.065109662 (102h34m32.628432666s) [noise=true , address=104.36.82.4:5808 , piece_id=JYCDN2XYIDVTQRXNTTNS2HKCKHHUCMZHMFPSEAQXZKRYQM2LMVLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.057635275 (102h34m39.62095826s) [noise=true , address=104.36.82.12:6172 , piece_id=MDYL4WPLGLA7RH4DVTU5XFNQNO6XUEZMDXQWCUPOZUIYJMCGWKJQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.275908809 (102h34m39.839231798s) [noise=true , address=104.36.82.54:4172 , piece_id=22HC55QPIEUNZTYBWKKJBIZSNPDY5Y4Y3PFZMPQUASVIOAALEZYA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:04.505169278 (102h34m40.068492264s) [noise=true , address=104.36.82.20:6465 , piece_id=LUBRO57XIVLAJ57MLL2EPE6PXSKE3JP4ZTEVZJV5ITJZJMT7BJIQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:10.345796585 (102h34m45.909119595s) [noise=true , address=104.36.82.15:6268 , piece_id=OQ4MJ2YD3ZDMAWMD2FYX5B7IT3FAV7H6E3XZUQKGJZTSCA3ILKJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:12.037082982 (102h34m47.600405983s) [noise=true , address=104.36.82.11:6118 , piece_id=7G7HKVPFY7QJUAIF42N6HXIDHPOXMDPC6WDMNQAAQIW7SGLYDDIA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:23.385517678 (102h34m58.94884067s) [noise=true , address=104.36.82.52:4096 , piece_id=MJ7RLJZEFNJKFCKHPCZXF4PE6U5LTTYV2XTGAUURYVL7QCWLNDRA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:31.933451504 (102h35m7.496774497s) [noise=true , address=104.36.82.55:4214 , piece_id=ROSO6MU6PIQ6VJWYK4CQELA442D2SNHVET2NJNPMYH56JPW44QJA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jun 22 22:58:40.800235456 (102h35m16.363558458s) [noise=true , address=104.36.82.14:6244 , piece_id=EA26GOEIWVNN3KFKTE4KEIQ7HFH665J5WLTOOOE7S7QNTJY5LJFQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:6244): protocol: expected piece hash; context deadline exceeded; EOF Jun 22 22:58:40.800270138 (102h35m16.363593144s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jun 22 22:58:40.830995933 (102h35m16.394318937s) [upload=82052]: Done uploading segments... Jun 22 22:58:40.831070435 (102h35m16.394393438s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 30.274473934s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X................X......................] duration: 17.740879564s FAIL (2025-06-22 22:59:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.740963169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X................X.....................] duration: 26.40644519s FAIL (2025-06-22 22:59:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.406472615s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)