Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X........X.X.X...XX.................X.X..........XX...X..] duration: 11.755287686s FAIL (2025-07-26 23:00:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.755314408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.............X................X........................] duration: 781.075341ms FAIL (2025-07-26 23:00:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 781.113459ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [....X........X.X.X...XX.................X.X..........XX...X.] duration: 7.129169372s FAIL (2025-07-26 22:59:57 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.129185492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.....X........X.X.X...XX.................X.X..........XX...X] duration: 1m0.000651635s FAIL (2025-07-26 22:59:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000688476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 187.435614ms FAIL (2025-07-26 22:59:49 UTC) orbiter warning: uplink: too many requests Duration: 187.476739ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X....X........X...........................................] duration: 1.757399725s FAIL (2025-07-26 22:59:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.75743928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.............X................X.......................] duration: 420.863457ms FAIL (2025-07-26 22:59:29 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 420.89406ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.X] duration: 1m0.034150021s FAIL (2025-07-26 22:59:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 88 (1h6m7.548390157s) [upload=1022]: Done uploading segments... Jul 26 22:57:51.767478006 (1h6m7.548411369s) [upload=1022]: Upload finished. err: <nil> Jul 26 22:58:28.502897453 (1h6m44.283830836s) [upload=1033]: Starting upload Jul 26 22:58:28.502997665 (1h6m44.283931137s) [upload=1033]: Done starting upload Jul 26 22:58:28.503232307 (1h6m44.284165689s) [upload=1033]: Uploading segments... Jul 26 22:58:28.503465067 (1h6m44.284398456s) [upload=1033]: Got next segment. Inline: false Jul 26 22:58:28.503615098 (1h6m44.284548479s) [upload=1033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:58:28.503623431 (1h6m44.284556812s) [upload=1033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:58:28.570016233 (1h6m44.3509496s) [seg_pos={0 0}, upload=1033]: Begin upload segment... Jul 26 22:58:28.570079800 (1h6m44.351013168s) [seg_pos={0 0}, upload=1033]: Done begin upload segment. Jul 26 22:58:28.570123784 (1h6m44.351057162s) [noise=true , address=77.90.31.13:4629 , piece_id=XK2HCP5FIW3T47WV5RHHP7BSXVYULNMTP6A2KSVKJTZNRM5KXLVQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.570677324 (1h6m44.351610711s) [noise=true , address=79.127.220.97:4361 , piece_id=GMU7F6FUM7YIQBBI6LG5YBQSJY42GKQ5N6FEIY4QPQLKBZYZCH5Q, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.571171134 (1h6m44.352104532s) [noise=true , address=45.91.116.9:4273 , piece_id=6JQNZBCWTDTSHXPOXRLXKFBFKQ4SQLRPRFCOGAAURAR4ISMKEL7Q, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.571685876 (1h6m44.352619264s) [noise=true , address=79.127.201.215:4359 , piece_id=YQ3AWQ2MRCTRCPUHB2MWVXBYDZP5EIUAJ3UDKZO5J5VREGWJBA5Q, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.572137096 (1h6m44.353070481s) [noise=true , address=77.90.31.10:4540 , piece_id=5A4ZJZGTMUBPTZE6DF7M4SSUOBRCVZPZIC5UFUTGIBNXFSBHUO4Q, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.572549346 (1h6m44.353482732s) [noise=true , address=79.127.219.47:4048 , piece_id=MSZTOXCHWLTLROZB5QS2WNQ7XYXDP5GPYVCLV3RMLNQVHUALUHYA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.572887011 (1h6m44.353820389s) [noise=true , address=79.127.219.38:4007 , piece_id=JACIQ7OSSWTUGPGIW3BMVXWGT3AFTLEVBNQM73N4JEXVNXCREFCQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.573219801 (1h6m44.354153185s) [noise=true , address=79.127.205.234:4616 , piece_id=YXP7BI3WRWSMIUZ32FZO25FMVIAIENXHNYWOCJD3KV4I3JSTKZSQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.573531179 (1h6m44.35446456s) [noise=true , address=64.49.13.140:4514 , piece_id=EBGC2D4HQK6JYC23HROMOTJWQDZCDHIJPR2BMQJ744SKYNL3CCPA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.573886217 (1h6m44.354819599s) [noise=true , address=79.127.205.235:4685 , piece_id=NK6LAY2CEHNFUCZWTIVNJ5XVXCXNS677L76M6PEVACPC2MSUKZYQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.574311436 (1h6m44.355244833s) [noise=true , address=79.127.205.246:4276 , piece_id=355QNCO2HK7RNGGXRXPMU6TM6NNXKC3DJFULNLD7PIABLIYEDXFA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.574807278 (1h6m44.355740672s) [noise=true , address=79.127.219.51:4212 , piece_id=VRTOZIPSPZOCY3POGZKE5NHFY3FP64WKKTS6LKUZKJVMENS7JPPQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.575296256 (1h6m44.356229641s) [noise=true , address=77.90.31.16:4767 , piece_id=YWOWZMTNGLCPX7BK4JFGWFOPE5MI73GZCKIVSU4Y7RJ2XCZ2HC3Q, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.575680350 (1h6m44.356613733s) [noise=true , address=79.127.219.48:4090 , piece_id=MN23T27ZNRREHVZKBPQTHEJTJFCPRIJV4EPSJQ6AW3HX6RQ67IIA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.576113708 (1h6m44.357047095s) [noise=true , address=79.127.205.245:4241 , piece_id=6MM7WUJU7UXSFBAMBOMBJRGYPZ6XBFEPKKLQZOGGMQAG3VJARRNA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.576504313 (1h6m44.357437692s) [noise=true , address=79.127.219.38:4018 , piece_id=HI24WLXWZU3SWVHVGYZKZERFEYLZHPESL6Z4YNPCRMRYK3NOJFJA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.576915348 (1h6m44.357848728s) [noise=true , address=79.127.205.233:4640 , piece_id=232XIHDK3YILEYWSS2KYCSIX6ZHLM6C6SMSFN2DJN65LOR6JOSYA, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.577280068 (1h6m44.358213449s) [noise=true , address=45.91.116.14:4222 , piece_id=CWQTATXU2KKEDLGEVUE7M6QGF436CASNMATUS45VTOLAWP6GQZPQ, seg_pos={0 0}, upload=1033]: Uploading piece... Jul 26 22:58:28.623880246 (1h6m44.404813609s) [noise=true , address=77.90.31.10:4540 , piece_id=5A4ZJZGTMUBPTZE6DF7M4SSUOBRCVZPZIC5UFUTGIBNXFSBHUO4Q, seg_pos={0 0}, upload=1033]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: storage node overloaded, request limit: 2000 Jul 26 22:58:28.632243188 (1h6m44.413176566s) [noise=true , address=79.127.219.47:4048 , piece_id=MSZTOXCHWLTLROZB5QS2WNQ7XYXDP5GPYVCLV3RMLNQVHUALUHYA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:58:28.671876505 (1h6m44.452809895s) [upload=1033]: Next returned nil segment Jul 26 22:58:28.671895946 (1h6m44.452829329s) [upload=1033]: Waiting for error group managing segments... Jul 26 22:58:28.816811973 (1h6m44.597745342s) [noise=true , address=45.91.116.14:4222 , piece_id=CWQTATXU2KKEDLGEVUE7M6QGF436CASNMATUS45VTOLAWP6GQZPQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.195730664 (1h6m44.976664033s) [noise=true , address=79.127.205.233:4640 , piece_id=232XIHDK3YILEYWSS2KYCSIX6ZHLM6C6SMSFN2DJN65LOR6JOSYA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.196412264 (1h6m44.977345632s) [noise=true , address=79.127.205.246:4276 , piece_id=355QNCO2HK7RNGGXRXPMU6TM6NNXKC3DJFULNLD7PIABLIYEDXFA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.196945070 (1h6m44.977878438s) [noise=true , address=79.127.205.234:4616 , piece_id=YXP7BI3WRWSMIUZ32FZO25FMVIAIENXHNYWOCJD3KV4I3JSTKZSQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.220488705 (1h6m45.00142208s) [noise=true , address=79.127.205.245:4241 , piece_id=6MM7WUJU7UXSFBAMBOMBJRGYPZ6XBFEPKKLQZOGGMQAG3VJARRNA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.387673191 (1h6m45.16860656s) [noise=true , address=79.127.205.235:4685 , piece_id=NK6LAY2CEHNFUCZWTIVNJ5XVXCXNS677L76M6PEVACPC2MSUKZYQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:29.703206499 (1h6m45.484139875s) [noise=true , address=79.127.219.38:4018 , piece_id=HI24WLXWZU3SWVHVGYZKZERFEYLZHPESL6Z4YNPCRMRYK3NOJFJA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:30.494455980 (1h6m46.275389348s) [noise=true , address=79.127.220.97:4361 , piece_id=GMU7F6FUM7YIQBBI6LG5YBQSJY42GKQ5N6FEIY4QPQLKBZYZCH5Q, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:30.666391987 (1h6m46.447325357s) [noise=true , address=79.127.219.51:4212 , piece_id=VRTOZIPSPZOCY3POGZKE5NHFY3FP64WKKTS6LKUZKJVMENS7JPPQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:30.802615577 (1h6m46.583548945s) [noise=true , address=79.127.219.38:4007 , piece_id=JACIQ7OSSWTUGPGIW3BMVXWGT3AFTLEVBNQM73N4JEXVNXCREFCQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:31.056423266 (1h6m46.837356644s) [noise=true , address=79.127.219.48:4090 , piece_id=MN23T27ZNRREHVZKBPQTHEJTJFCPRIJV4EPSJQ6AW3HX6RQ67IIA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:31.338443482 (1h6m47.119376859s) [noise=true , address=64.49.13.140:4514 , piece_id=EBGC2D4HQK6JYC23HROMOTJWQDZCDHIJPR2BMQJ744SKYNL3CCPA, seg_pos={0 0}, upload=1033]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 22:58:36.604546701 (1h6m52.385480072s) [noise=true , address=79.127.201.215:4359 , piece_id=YQ3AWQ2MRCTRCPUHB2MWVXBYDZP5EIUAJ3UDKZO5J5VREGWJBA5Q, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:58:36.698911243 (1h6m52.47984461s) [noise=true , address=45.91.116.9:4273 , piece_id=6JQNZBCWTDTSHXPOXRLXKFBFKQ4SQLRPRFCOGAAURAR4ISMKEL7Q, seg_pos={0 0}, upload=1033]: Done uploading piece. err: <nil> Jul 26 22:59:28.471592863 (1h7m44.252526228s) [noise=true , address=77.90.31.13:4629 , piece_id=XK2HCP5FIW3T47WV5RHHP7BSXVYULNMTP6A2KSVKJTZNRM5KXLVQ, seg_pos={0 0}, upload=1033]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: write tcp 159.223.143.39:38664->77.90.31.13:4629: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:28.471647640 (1h7m44.252581009s) [noise=true , address=77.90.31.16:4767 , piece_id=YWOWZMTNGLCPX7BK4JFGWFOPE5MI73GZCKIVSU4Y7RJ2XCZ2HC3Q, seg_pos={0 0}, upload=1033]: Done uploading piece. err: ecclient: upload failed (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK, address:77.90.31.16:4767): protocol: write tcp 159.223.143.39:38400->77.90.31.16:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:59:28.471669232 (1h7m44.252602609s) [seg_pos={0 0}, upload=1033]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:59:28.501757396 (1h7m44.282690774s) [upload=1033]: Done uploading segments... Jul 26 22:59:28.501797672 (1h7m44.282731048s) [upload=1033]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.034196505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 185.258628ms FAIL (2025-07-26 22:59:17 UTC) orbiter warning: uplink: too many requests Duration: 185.286159ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.............X................X......................] duration: 1.837879948s FAIL (2025-07-26 22:59:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.837909093s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 219.930396ms FAIL (2025-07-26 22:58:58 UTC) orbiter warning: uplink: too many requests Duration: 219.975415ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X....X........X..........................................] duration: 416.964025ms FAIL (2025-07-26 22:58:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 416.984116ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X......X.............X................X.....................] duration: 1.918305719s FAIL (2025-07-26 22:58:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.918337159s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 197.956293ms FAIL (2025-07-26 22:58:22 UTC) orbiter warning: uplink: too many requests Duration: 197.98939ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 195.120013ms FAIL (2025-07-26 22:57:59 UTC) orbiter warning: uplink: too many requests Duration: 195.150524ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [......X........X.X.X...XX.................X.X..........XX...] duration: 11.161040094s FAIL (2025-07-26 22:57:57 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.161059266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX.] duration: 5.782457085s FAIL (2025-07-26 22:57:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== .505442529s) [upload=1003]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:56:32.761319638 (1h4m48.542253027s) [seg_pos={0 0}, upload=1003]: Begin upload segment... Jul 26 22:56:32.761432974 (1h4m48.542366355s) [seg_pos={0 0}, upload=1003]: Done begin upload segment. Jul 26 22:56:32.761477820 (1h4m48.542411202s) [noise=true , address=79.127.220.101:4048 , piece_id=TVP345FFF2K6UEF2H22BEVWU3O5WH3XDGTKDTUDKYD67OG7O67QA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762109645 (1h4m48.543043034s) [noise=true , address=77.90.31.16:4746 , piece_id=WMKPRDVL2SEZJ3UJEQHBIH2JS2JY66ZNUH2XGFRZQDCM2T6OYHWQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762361611 (1h4m48.543294999s) [noise=true , address=64.49.13.170:4719 , piece_id=LCPCJJ7AYZKDGP5JAY4MLJEJO6ZJJHXXLD3Q3KTSSIEMLDLHUARA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762780632 (1h4m48.543714033s) [noise=true , address=77.90.31.17:4782 , piece_id=5XFMV3EQHUWELC6M7MZQHA7ETAJI3EI6FOZGJ4M5D3DGZ2YV7LFQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762908017 (1h4m48.543841405s) [noise=true , address=79.127.201.215:4350 , piece_id=6K564YWYL4DCFPKMIDIOI7ND5OPD2EJCDTMIBNLBHZQXDWIRW2PQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763409578 (1h4m48.544342966s) [noise=true , address=64.49.13.169:4716 , piece_id=EIOMDZATNT7APLWUNT2VBA76EUNPT477USSBDDT7HECUQRD6BPMQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763531573 (1h4m48.544464957s) [noise=true , address=79.127.205.246:4275 , piece_id=ZOLLIB2TPN46WHUMTOWKQL6UGXEC5MESP7KKHCTLOJPAAB5MGT3A, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763877060 (1h4m48.544810515s) [noise=true , address=156.146.43.225:4448 , piece_id=DYKB22KF4QIW3RDZJCDZMXMZ3TV6DPYZFPTUBBNJ3H6GXD3XA66A, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764091544 (1h4m48.545024933s) [noise=true , address=45.91.116.9:4288 , piece_id=TA7RSWL4GOEOZ3JWXCE62BKOZLMRHLGKN5SGWT4LC3PTZSLXE5NQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764412768 (1h4m48.545346156s) [noise=true , address=79.127.201.215:4345 , piece_id=QBREW3FB467WCTFL5MB6DPWQXHZVUGBE2PV2LQYZU47AO7B3UTPA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764678280 (1h4m48.54561167s) [noise=true , address=45.91.116.14:4222 , piece_id=BOBNQUQNXQCJRZWZAPQ5GCLQO73Y4437QM7JY66RIU76TBV6ZCNA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764860900 (1h4m48.545794285s) [noise=true , address=45.91.116.9:4277 , piece_id=ZRAUWEPI7RWSZ27LW4IEEGLCWQPVPKBUIY24DUDUTYE6VSJUA24Q, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765166007 (1h4m48.546099397s) [noise=true , address=77.90.31.10:4540 , piece_id=ZP3GZOMNTLQ23WNDVUTDRXCLWZ6WTJLBEVBA4C7OW5N37KCFHKDA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765283087 (1h4m48.546216475s) [noise=true , address=79.127.205.245:4251 , piece_id=NERWPNU2KEMQZNPDZLSYNNX2J25H4S4HT4ZAPZW652QYMQHJ2WBQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765695313 (1h4m48.546628702s) [noise=true , address=79.127.219.51:4210 , piece_id=E3O7WN4MJXAPYTJ3JBS2QB5DPIEFN2SBPKSWZQDXLP2DNY4MDLDA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766142042 (1h4m48.547075431s) [noise=true , address=45.91.116.5:4310 , piece_id=YZ6ULUMBX7YOLR6KVMG5I4RQXGPL3USYJ547GCJEYIWJX6DKEIKA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766570851 (1h4m48.54750424s) [noise=true , address=77.90.31.11:4581 , piece_id=6TPSO2XZ7GPQ46ZMIEUXITWLMQAMYNOYDB3XWUEX6LLUREBCUULA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766997991 (1h4m48.547931379s) [noise=true , address=45.91.116.11:4151 , piece_id=5MV24DI4E3PR4SFDP5RIK4L3OLBZQC635TU3QHPWDLU3JAKO3XNA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.812919497 (1h4m48.593852878s) [noise=true , address=77.90.31.10:4540 , piece_id=ZP3GZOMNTLQ23WNDVUTDRXCLWZ6WTJLBEVBA4C7OW5N37KCFHKDA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:32.823436329 (1h4m48.604369716s) [noise=true , address=45.91.116.9:4277 , piece_id=ZRAUWEPI7RWSZ27LW4IEEGLCWQPVPKBUIY24DUDUTYE6VSJUA24Q, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:32.855001154 (1h4m48.635934542s) [upload=1003]: Next returned nil segment Jul 26 22:56:32.855041745 (1h4m48.635975129s) [upload=1003]: Waiting for error group managing segments... Jul 26 22:56:32.915064007 (1h4m48.695997397s) [noise=true , address=77.90.31.11:4581 , piece_id=6TPSO2XZ7GPQ46ZMIEUXITWLMQAMYNOYDB3XWUEX6LLUREBCUULA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:32.975276655 (1h4m48.756210041s) [noise=true , address=79.127.220.101:4048 , piece_id=TVP345FFF2K6UEF2H22BEVWU3O5WH3XDGTKDTUDKYD67OG7O67QA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:33.373868717 (1h4m49.154802094s) [noise=true , address=79.127.205.245:4251 , piece_id=NERWPNU2KEMQZNPDZLSYNNX2J25H4S4HT4ZAPZW652QYMQHJ2WBQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:33.378685049 (1h4m49.159618427s) [noise=true , address=79.127.205.246:4275 , piece_id=ZOLLIB2TPN46WHUMTOWKQL6UGXEC5MESP7KKHCTLOJPAAB5MGT3A, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:34.084339504 (1h4m49.865272884s) [noise=true , address=45.91.116.9:4288 , piece_id=TA7RSWL4GOEOZ3JWXCE62BKOZLMRHLGKN5SGWT4LC3PTZSLXE5NQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:4288): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.423681143 (1h4m51.204614521s) [noise=true , address=45.91.116.5:4310 , piece_id=YZ6ULUMBX7YOLR6KVMG5I4RQXGPL3USYJ547GCJEYIWJX6DKEIKA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.569580434 (1h4m51.350513818s) [noise=true , address=64.49.13.169:4716 , piece_id=EIOMDZATNT7APLWUNT2VBA76EUNPT477USSBDDT7HECUQRD6BPMQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.817104985 (1h4m51.598038363s) [noise=true , address=156.146.43.225:4448 , piece_id=DYKB22KF4QIW3RDZJCDZMXMZ3TV6DPYZFPTUBBNJ3H6GXD3XA66A, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:35.855907345 (1h4m51.636840735s) [noise=true , address=79.127.219.51:4210 , piece_id=E3O7WN4MJXAPYTJ3JBS2QB5DPIEFN2SBPKSWZQDXLP2DNY4MDLDA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:38.925099113 (1h4m54.706032491s) [noise=true , address=64.49.13.170:4719 , piece_id=LCPCJJ7AYZKDGP5JAY4MLJEJO6ZJJHXXLD3Q3KTSSIEMLDLHUARA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:40.437355674 (1h4m56.218289047s) [noise=true , address=45.91.116.11:4151 , piece_id=5MV24DI4E3PR4SFDP5RIK4L3OLBZQC635TU3QHPWDLU3JAKO3XNA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:54.504735029 (1h5m10.28566842s) [noise=true , address=45.91.116.14:4222 , piece_id=BOBNQUQNXQCJRZWZAPQ5GCLQO73Y4437QM7JY66RIU76TBV6ZCNA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:20.141554727 (1h5m35.922488104s) [noise=true , address=79.127.201.215:4345 , piece_id=QBREW3FB467WCTFL5MB6DPWQXHZVUGBE2PV2LQYZU47AO7B3UTPA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:57:26.897069539 (1h5m42.67800291s) [noise=true , address=79.127.201.215:4350 , piece_id=6K564YWYL4DCFPKMIDIOI7ND5OPD2EJCDTMIBNLBHZQXDWIRW2PQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:57:32.688562247 (1h5m48.469495608s) [noise=true , address=77.90.31.17:4782 , piece_id=5XFMV3EQHUWELC6M7MZQHA7ETAJI3EI6FOZGJ4M5D3DGZ2YV7LFQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, address:77.90.31.17:4782): protocol: write tcp 159.223.143.39:43290->77.90.31.17:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:57:32.688601675 (1h5m48.469535037s) [noise=true , address=77.90.31.16:4746 , piece_id=WMKPRDVL2SEZJ3UJEQHBIH2JS2JY66ZNUH2XGFRZQDCM2T6OYHWQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: read tcp 159.223.143.39:35968->77.90.31.16:4746: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:57:32.688620974 (1h5m48.469554349s) [seg_pos={0 0}, upload=1003]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:57:32.714574471 (1h5m48.49550784s) [upload=1003]: Done uploading segments... Jul 26 22:57:32.714611929 (1h5m48.495545305s) [upload=1003]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration:* 5.782487017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X......X.............X................X....................] duration: 464.375623ms FAIL (2025-07-26 22:57:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 464.410974ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXXX] duration: 1m0.028842718s FAIL (2025-07-26 22:57:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== .505442529s) [upload=1003]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:56:32.761319638 (1h4m48.542253027s) [seg_pos={0 0}, upload=1003]: Begin upload segment... Jul 26 22:56:32.761432974 (1h4m48.542366355s) [seg_pos={0 0}, upload=1003]: Done begin upload segment. Jul 26 22:56:32.761477820 (1h4m48.542411202s) [noise=true , address=79.127.220.101:4048 , piece_id=TVP345FFF2K6UEF2H22BEVWU3O5WH3XDGTKDTUDKYD67OG7O67QA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762109645 (1h4m48.543043034s) [noise=true , address=77.90.31.16:4746 , piece_id=WMKPRDVL2SEZJ3UJEQHBIH2JS2JY66ZNUH2XGFRZQDCM2T6OYHWQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762361611 (1h4m48.543294999s) [noise=true , address=64.49.13.170:4719 , piece_id=LCPCJJ7AYZKDGP5JAY4MLJEJO6ZJJHXXLD3Q3KTSSIEMLDLHUARA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762780632 (1h4m48.543714033s) [noise=true , address=77.90.31.17:4782 , piece_id=5XFMV3EQHUWELC6M7MZQHA7ETAJI3EI6FOZGJ4M5D3DGZ2YV7LFQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.762908017 (1h4m48.543841405s) [noise=true , address=79.127.201.215:4350 , piece_id=6K564YWYL4DCFPKMIDIOI7ND5OPD2EJCDTMIBNLBHZQXDWIRW2PQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763409578 (1h4m48.544342966s) [noise=true , address=64.49.13.169:4716 , piece_id=EIOMDZATNT7APLWUNT2VBA76EUNPT477USSBDDT7HECUQRD6BPMQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763531573 (1h4m48.544464957s) [noise=true , address=79.127.205.246:4275 , piece_id=ZOLLIB2TPN46WHUMTOWKQL6UGXEC5MESP7KKHCTLOJPAAB5MGT3A, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.763877060 (1h4m48.544810515s) [noise=true , address=156.146.43.225:4448 , piece_id=DYKB22KF4QIW3RDZJCDZMXMZ3TV6DPYZFPTUBBNJ3H6GXD3XA66A, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764091544 (1h4m48.545024933s) [noise=true , address=45.91.116.9:4288 , piece_id=TA7RSWL4GOEOZ3JWXCE62BKOZLMRHLGKN5SGWT4LC3PTZSLXE5NQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764412768 (1h4m48.545346156s) [noise=true , address=79.127.201.215:4345 , piece_id=QBREW3FB467WCTFL5MB6DPWQXHZVUGBE2PV2LQYZU47AO7B3UTPA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764678280 (1h4m48.54561167s) [noise=true , address=45.91.116.14:4222 , piece_id=BOBNQUQNXQCJRZWZAPQ5GCLQO73Y4437QM7JY66RIU76TBV6ZCNA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.764860900 (1h4m48.545794285s) [noise=true , address=45.91.116.9:4277 , piece_id=ZRAUWEPI7RWSZ27LW4IEEGLCWQPVPKBUIY24DUDUTYE6VSJUA24Q, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765166007 (1h4m48.546099397s) [noise=true , address=77.90.31.10:4540 , piece_id=ZP3GZOMNTLQ23WNDVUTDRXCLWZ6WTJLBEVBA4C7OW5N37KCFHKDA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765283087 (1h4m48.546216475s) [noise=true , address=79.127.205.245:4251 , piece_id=NERWPNU2KEMQZNPDZLSYNNX2J25H4S4HT4ZAPZW652QYMQHJ2WBQ, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.765695313 (1h4m48.546628702s) [noise=true , address=79.127.219.51:4210 , piece_id=E3O7WN4MJXAPYTJ3JBS2QB5DPIEFN2SBPKSWZQDXLP2DNY4MDLDA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766142042 (1h4m48.547075431s) [noise=true , address=45.91.116.5:4310 , piece_id=YZ6ULUMBX7YOLR6KVMG5I4RQXGPL3USYJ547GCJEYIWJX6DKEIKA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766570851 (1h4m48.54750424s) [noise=true , address=77.90.31.11:4581 , piece_id=6TPSO2XZ7GPQ46ZMIEUXITWLMQAMYNOYDB3XWUEX6LLUREBCUULA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.766997991 (1h4m48.547931379s) [noise=true , address=45.91.116.11:4151 , piece_id=5MV24DI4E3PR4SFDP5RIK4L3OLBZQC635TU3QHPWDLU3JAKO3XNA, seg_pos={0 0}, upload=1003]: Uploading piece... Jul 26 22:56:32.812919497 (1h4m48.593852878s) [noise=true , address=77.90.31.10:4540 , piece_id=ZP3GZOMNTLQ23WNDVUTDRXCLWZ6WTJLBEVBA4C7OW5N37KCFHKDA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:32.823436329 (1h4m48.604369716s) [noise=true , address=45.91.116.9:4277 , piece_id=ZRAUWEPI7RWSZ27LW4IEEGLCWQPVPKBUIY24DUDUTYE6VSJUA24Q, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:32.855001154 (1h4m48.635934542s) [upload=1003]: Next returned nil segment Jul 26 22:56:32.855041745 (1h4m48.635975129s) [upload=1003]: Waiting for error group managing segments... Jul 26 22:56:32.915064007 (1h4m48.695997397s) [noise=true , address=77.90.31.11:4581 , piece_id=6TPSO2XZ7GPQ46ZMIEUXITWLMQAMYNOYDB3XWUEX6LLUREBCUULA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:32.975276655 (1h4m48.756210041s) [noise=true , address=79.127.220.101:4048 , piece_id=TVP345FFF2K6UEF2H22BEVWU3O5WH3XDGTKDTUDKYD67OG7O67QA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:33.373868717 (1h4m49.154802094s) [noise=true , address=79.127.205.245:4251 , piece_id=NERWPNU2KEMQZNPDZLSYNNX2J25H4S4HT4ZAPZW652QYMQHJ2WBQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:33.378685049 (1h4m49.159618427s) [noise=true , address=79.127.205.246:4275 , piece_id=ZOLLIB2TPN46WHUMTOWKQL6UGXEC5MESP7KKHCTLOJPAAB5MGT3A, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:34.084339504 (1h4m49.865272884s) [noise=true , address=45.91.116.9:4288 , piece_id=TA7RSWL4GOEOZ3JWXCE62BKOZLMRHLGKN5SGWT4LC3PTZSLXE5NQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:4288): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.423681143 (1h4m51.204614521s) [noise=true , address=45.91.116.5:4310 , piece_id=YZ6ULUMBX7YOLR6KVMG5I4RQXGPL3USYJ547GCJEYIWJX6DKEIKA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.569580434 (1h4m51.350513818s) [noise=true , address=64.49.13.169:4716 , piece_id=EIOMDZATNT7APLWUNT2VBA76EUNPT477USSBDDT7HECUQRD6BPMQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:35.817104985 (1h4m51.598038363s) [noise=true , address=156.146.43.225:4448 , piece_id=DYKB22KF4QIW3RDZJCDZMXMZ3TV6DPYZFPTUBBNJ3H6GXD3XA66A, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:35.855907345 (1h4m51.636840735s) [noise=true , address=79.127.219.51:4210 , piece_id=E3O7WN4MJXAPYTJ3JBS2QB5DPIEFN2SBPKSWZQDXLP2DNY4MDLDA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:38.925099113 (1h4m54.706032491s) [noise=true , address=64.49.13.170:4719 , piece_id=LCPCJJ7AYZKDGP5JAY4MLJEJO6ZJJHXXLD3Q3KTSSIEMLDLHUARA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:40.437355674 (1h4m56.218289047s) [noise=true , address=45.91.116.11:4151 , piece_id=5MV24DI4E3PR4SFDP5RIK4L3OLBZQC635TU3QHPWDLU3JAKO3XNA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:56:54.504735029 (1h5m10.28566842s) [noise=true , address=45.91.116.14:4222 , piece_id=BOBNQUQNXQCJRZWZAPQ5GCLQO73Y4437QM7JY66RIU76TBV6ZCNA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:57:20.141554727 (1h5m35.922488104s) [noise=true , address=79.127.201.215:4345 , piece_id=QBREW3FB467WCTFL5MB6DPWQXHZVUGBE2PV2LQYZU47AO7B3UTPA, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:57:26.897069539 (1h5m42.67800291s) [noise=true , address=79.127.201.215:4350 , piece_id=6K564YWYL4DCFPKMIDIOI7ND5OPD2EJCDTMIBNLBHZQXDWIRW2PQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: <nil> Jul 26 22:57:32.688562247 (1h5m48.469495608s) [noise=true , address=77.90.31.17:4782 , piece_id=5XFMV3EQHUWELC6M7MZQHA7ETAJI3EI6FOZGJ4M5D3DGZ2YV7LFQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, address:77.90.31.17:4782): protocol: write tcp 159.223.143.39:43290->77.90.31.17:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:57:32.688601675 (1h5m48.469535037s) [noise=true , address=77.90.31.16:4746 , piece_id=WMKPRDVL2SEZJ3UJEQHBIH2JS2JY66ZNUH2XGFRZQDCM2T6OYHWQ, seg_pos={0 0}, upload=1003]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: read tcp 159.223.143.39:35968->77.90.31.16:4746: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:57:32.688620974 (1h5m48.469554349s) [seg_pos={0 0}, upload=1003]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:57:32.714574471 (1h5m48.49550784s) [upload=1003]: Done uploading segments... Jul 26 22:57:32.714611929 (1h5m48.495545305s) [upload=1003]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration:* 1m0.028865819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X....X........X.........................................] duration: 577.368757ms FAIL (2025-07-26 22:57:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 577.401149ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....X....X........X........................................] duration: 525.07917ms FAIL (2025-07-26 22:57:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 525.11794ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X......X.............X................X...................] duration: 418.777523ms FAIL (2025-07-26 22:56:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 418.810212ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 228.648159ms FAIL (2025-07-26 22:56:45 UTC) orbiter warning: uplink: too many requests Duration: 228.669936ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.......X........X.X.X...XX.................X.X..........XX..] duration: 6.612994275s FAIL (2025-07-26 22:56:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.613103945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XXX] duration: 1m0.03039379s FAIL (2025-07-26 22:56:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== NA, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:55:32.674243598 (1h3m48.455176986s) [noise=true , address=79.127.205.246:4286 , piece_id=OQA4AIYP5YZ3F477SPF6GNBMM7SHV5U3KATHY6QQJ3WIBP7CO5EA, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:55:32.677985321 (1h3m48.458918697s) [noise=true , address=77.90.31.21:4907 , piece_id=PXE3CKSU44PJHR5UK5LYIH6VB6HGMLMF635TEXWNZJCSHPLJKN4A, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:55:32.723216630 (1h3m48.504150019s) [noise=true , address=77.90.31.16:4746 , piece_id=I5VVF4LOTMJACG24R6NCQFJITFOGPVQ6DHPHDKNLFEWGZJJIXFOQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:32.771831145 (1h3m48.55276453s) [upload=985]: Next returned nil segment Jul 26 22:55:32.771853345 (1h3m48.552786728s) [upload=985]: Waiting for error group managing segments... Jul 26 22:55:32.807532859 (1h3m48.588466237s) [noise=true , address=77.90.31.21:4907 , piece_id=PXE3CKSU44PJHR5UK5LYIH6VB6HGMLMF635TEXWNZJCSHPLJKN4A, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:32.880403589 (1h3m48.66133698s) [noise=true , address=79.127.219.47:4061 , piece_id=6OXKEMPPAQDVFFVMVW7CXA7KRJVYCIZZEHXUOSC7V5KRTUZIQRWQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:33.255832809 (1h3m49.036766172s) [noise=true , address=79.127.205.246:4286 , piece_id=OQA4AIYP5YZ3F477SPF6GNBMM7SHV5U3KATHY6QQJ3WIBP7CO5EA, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:33.263794266 (1h3m49.04472763s) [noise=true , address=79.127.205.235:4684 , piece_id=HS7OT27GKKLBDVERCH2D5JHO3SC5SKEITMV3YGFCCMCSWLDI7WYQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:33.281905164 (1h3m49.062838532s) [noise=true , address=79.127.205.233:4654 , piece_id=H653RFI273YSI5OZMHNDNRPHZRROJYHQP5QQQVND2DFGQ6OWYQZA, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:33.302210041 (1h3m49.083143411s) [noise=true , address=79.127.205.233:4633 , piece_id=X5KEHXPXQ5ZHSBWZRI4QKRUQAKEWDI3E4DTIZ7KQRQGCO7YSLENA, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:33.450651005 (1h3m49.231584396s) [noise=true , address=64.49.13.140:4514 , piece_id=3JKSLMMZAQIGXDR4KGKFUHKCDOKM4JOZZXWM2M2MDUQJNFDGTODQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:33.835499140 (1h3m49.616432517s) [noise=true , address=64.49.13.140:4508 , piece_id=UY7JZETDWQBMAD3IC5KSAKTZM2BJZQJNLHGC4JA64FSSH2IC2TNQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:34.505250520 (1h3m50.286183888s) [noise=true , address=64.49.13.138:4483 , piece_id=HWXDZN4M4MAEN46SABLZTZ2GT2PT4LLGYYWLWW64OBULBITAL54Q, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:36.438194749 (1h3m52.219128113s) [noise=true , address=79.127.220.101:4048 , piece_id=HMFY7FMMOOEKR26BYPGMTMCGK2BIRVVILNX54B6DGVEWYL66LWQQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:38.274315868 (1h3m54.055249252s) [noise=true , address=45.91.116.10:4100 , piece_id=QNYKZFKTJODQWMQUCTEYTKDDZXTB7F7UB7K5HM7MWB4HOS77ZEVA, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:40.170802737 (1h3m55.951736119s) [noise=true , address=45.91.116.14:4206 , piece_id=7O3U4KDLNN5TBQBCVAVA2ER2TFICPV3CKVBVWLLLWVDRIJJVKKWQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:55:43.140523230 (1h3m58.921456601s) [noise=true , address=64.49.13.170:4721 , piece_id=LVXHL2NSU2OR53P4GALQMQUGXNRDKIPMMBEE2YI4HIYMOFOR66CQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:49.679945213 (1h4m5.460878582s) [noise=true , address=45.91.116.11:4151 , piece_id=7SAM6RVHGM4DFEHMKPPSQG7PZ3F5FSMA4DV2U5OPOMMY3HYVCRPQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:55:59.189919044 (1h4m14.970852414s) [noise=true , address=45.91.116.11:4121 , piece_id=JXUIGBMQXDYD2BTPDHSAFWSNTK56UQXDKKDTSM2SQLMNMPR3YG7Q, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:56:04.555829348 (1h4m20.336762749s) [noise=true , address=79.127.201.215:4356 , piece_id=LBYZQI5UZGPQ5JCY4SY4CEFQB65VPLJVAOXKGJ3NYFJJJ3Q2YMBQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:56:09.533009245 (1h4m25.313942612s) [noise=true , address=79.127.201.215:4344 , piece_id=LFZ4KAK6673NULXX7GW2SAM7X5S52OLYYTLJ6TV5CS4O3K4Y375A, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:56:09.553183918 (1h4m25.334117304s) [noise=true , address=79.127.219.48:4090 , piece_id=H4MW7BEMACRS5JNOD3XCJNOQGPOI6IM2RC245L2VCU5JB5RY4FYQ, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.553384364 (1h4m25.334317744s) [noise=true , address=79.127.219.47:4061 , piece_id=ZQYG3BYQYAPVTCFEHL7EQUYNT5SRY42MLBCAR3RNJQBXVUCAAP2A, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.553547502 (1h4m25.334480872s) [noise=true , address=77.90.31.16:4754 , piece_id=XKHYQ5BIWB6XWGP2WYQUHIO445Z2JA5ZW6UXCNXMUBHSK263IVOA, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.556202697 (1h4m25.337136078s) [noise=true , address=79.127.220.101:4048 , piece_id=TZIJSEQEPNAPVOAW23HVSVEXKSB63USB7CXIZLKI4M6PKZJXYIZA, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.558563432 (1h4m25.339496809s) [noise=true , address=79.127.219.51:4212 , piece_id=WJNITFJHCAOEV2JL4ZFOO5LAJWXA5RXLOUDKCBJPKTFB7QWGYGJQ, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.561012107 (1h4m25.341945483s) [noise=true , address=79.127.220.97:4361 , piece_id=X6RK4O5Z7DAUSCWKGM32PPAAHDUKLULIEBHSHHIL5HEB4GTUNMNQ, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.563700684 (1h4m25.344634049s) [noise=true , address=79.127.201.215:4357 , piece_id=7WTEARAO64KCKVWO7RAN6IWMIV4XKYKFHMSCTYHWZTTIRCJHVUSQ, seg_pos={0 0}, upload=985]: Uploading piece... Jul 26 22:56:09.568467582 (1h4m25.349400958s) [noise=true , address=79.127.219.47:4061 , piece_id=ZQYG3BYQYAPVTCFEHL7EQUYNT5SRY42MLBCAR3RNJQBXVUCAAP2A, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:09.659522614 (1h4m25.440455981s) [noise=true , address=79.127.220.97:4361 , piece_id=X6RK4O5Z7DAUSCWKGM32PPAAHDUKLULIEBHSHHIL5HEB4GTUNMNQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:09.694937995 (1h4m25.475871403s) [noise=true , address=79.127.201.215:4357 , piece_id=7WTEARAO64KCKVWO7RAN6IWMIV4XKYKFHMSCTYHWZTTIRCJHVUSQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1yXKsyPEVG4W52L7SUmsDGt4oD2D9YyqsfwDVWpjMnMnyVo2Da, address:79.127.201.215:4357): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:09.700602365 (1h4m25.481535748s) [noise=true , address=79.127.220.101:4048 , piece_id=TZIJSEQEPNAPVOAW23HVSVEXKSB63USB7CXIZLKI4M6PKZJXYIZA, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: expected piece hash; storage node overloaded, request limit: 2000; EOF Jul 26 22:56:09.974242450 (1h4m25.755175837s) [noise=true , address=79.127.219.51:4212 , piece_id=WJNITFJHCAOEV2JL4ZFOO5LAJWXA5RXLOUDKCBJPKTFB7QWGYGJQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:56:10.430438521 (1h4m26.211371893s) [noise=true , address=79.127.219.48:4090 , piece_id=H4MW7BEMACRS5JNOD3XCJNOQGPOI6IM2RC245L2VCU5JB5RY4FYQ, seg_pos={0 0}, upload=985]: Done uploading piece. err: <nil> Jul 26 22:56:32.588868610 (1h4m48.369802037s) [noise=true , address=77.90.31.16:4754 , piece_id=XKHYQ5BIWB6XWGP2WYQUHIO445Z2JA5ZW6UXCNXMUBHSK263IVOA, seg_pos={0 0}, upload=985]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: read tcp 159.223.143.39:57244->77.90.31.16:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:56:32.588907708 (1h4m48.369841096s) [seg_pos={0 0}, upload=985]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:56:32.615742699 (1h4m48.396676069s) [upload=985]: Done uploading segments... Jul 26 22:56:32.615778212 (1h4m48.396711598s) [upload=985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.030419854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....X....X........X.......................................] duration: 467.482785ms FAIL (2025-07-26 22:56:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.559832ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X......X.............X................X..................] duration: 477.797089ms FAIL (2025-07-26 22:56:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 477.82976ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 195.180727ms FAIL (2025-07-26 22:56:18 UTC) orbiter warning: uplink: too many requests Duration: 195.273008ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [........X........X.X.X...XX.................X.X..........XX.] duration: 9.708611643s FAIL (2025-07-26 22:56:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.708629063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.........X........X.X.X...XX.................X.X..........XX] duration: 1m0.00062972s FAIL (2025-07-26 22:56:07 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000642837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X......X.............X................X.................] duration: 766.413801ms FAIL (2025-07-26 22:56:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 766.457054ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X....X....X........X......................................] duration: 519.204331ms FAIL (2025-07-26 22:55:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 519.238337ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XX] duration: 1m0.01703931s FAIL (2025-07-26 22:55:32 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== gment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:54:32.438371519 (1h2m48.219304899s) [upload=955]: Done uploading segments... Jul 26 22:54:32.438402298 (1h2m48.219335675s) [upload=955]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:54:32.548997773 (1h2m48.329931151s) [upload=971]: Starting upload Jul 26 22:54:32.549112830 (1h2m48.330046211s) [upload=971]: Done starting upload Jul 26 22:54:32.549220419 (1h2m48.330153799s) [upload=971]: Uploading segments... Jul 26 22:54:32.549570455 (1h2m48.33050384s) [upload=971]: Got next segment. Inline: false Jul 26 22:54:32.549633930 (1h2m48.33056731s) [upload=971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:54:32.549640869 (1h2m48.330574247s) [upload=971]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:54:32.597154071 (1h2m48.37808744s) [seg_pos={0 0}, upload=971]: Begin upload segment... Jul 26 22:54:32.597256302 (1h2m48.378189667s) [seg_pos={0 0}, upload=971]: Done begin upload segment. Jul 26 22:54:32.597289071 (1h2m48.378222452s) [noise=true , address=45.91.116.8:4057 , piece_id=FCDXB7XKJYCZ3HFBISW5KLGGWN57S7XFBBPUHEQJEVNVVLY7CZ3A, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.597325403 (1h2m48.378258794s) [noise=true , address=79.127.205.234:4605 , piece_id=E2GCBR5YSDUD4EZFLSF6GWVU6H7ZXWSQQTDQRO3C5HDCTOZ6FLFA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.597826029 (1h2m48.378759419s) [noise=true , address=77.90.31.21:4885 , piece_id=KEI7M7TMNBHZYEDH3IK2KUXCONNYTDZEXZIFOQ4HOYMV7E4LITTQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.597837835 (1h2m48.3787712s) [noise=true , address=77.90.31.11:4585 , piece_id=DQKVLVHJ4LOSIKDY73DVPDU2DNKCINTGZ4E7ZXN5YWE2LWSHMYMQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.598279837 (1h2m48.379213226s) [noise=true , address=79.127.201.215:4359 , piece_id=BZPNWTVQDFBF4QQM2JBT5RULPCHKCCKRDD365EVQZVMQJCIQIIDA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.598297728 (1h2m48.379231111s) [noise=true , address=64.49.13.140:4508 , piece_id=5PTUXJK6LL3GHBIJROFHT4RXJXBINNQC3M662JSKJLUTEQDCGWEA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.598723772 (1h2m48.379657152s) [noise=true , address=45.91.116.14:4194 , piece_id=EQUU5VQL7BSI6IS5KK4JLLA6DRP4O6BJYVWMAHA24RH4DJNDAO3Q, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.598739114 (1h2m48.379672503s) [noise=true , address=79.127.220.97:4361 , piece_id=WQ3QJTHWC5ALCQZNGO7FB45HRRZW55XHKRTQRXHQFB7NJ7Y73F7A, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.599172099 (1h2m48.380105485s) [noise=true , address=45.91.116.12:4229 , piece_id=HXNEO7TB6ZA2ALJAX25WTT4DRSQBKGDVRY57WNGZT6CB4LGB3CMA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.599238563 (1h2m48.380171966s) [noise=true , address=79.127.219.47:4048 , piece_id=UC37CIEOV35UCVM5MHFH23R5OFN47SLYKZMXRP66JHHNBE5D3OTA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.599631938 (1h2m48.380565321s) [noise=true , address=79.127.201.215:4346 , piece_id=AFCOK2MOOS4ZTPLHWFICB5KHE7PZKMIXK2J3RAPSUUDLUSSO5RBQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.599653524 (1h2m48.380586914s) [noise=true , address=45.91.116.9:4267 , piece_id=OE57ZC2HOABRYXKTQSFTOVQ7TQ7VHUSITVOMMFKGMPHSQ3XCRPHQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.600093317 (1h2m48.381026707s) [noise=true , address=79.127.205.245:4244 , piece_id=EYN5ZWSWAXAKOGVRJHJXBESOPUG36M7SQXV6ZGB7LCLPDHU6JYNQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.600119131 (1h2m48.381052509s) [noise=true , address=77.90.31.13:4644 , piece_id=Y4D4T3SFZTF7NJJWPOMJLHOK23CVC7ZL3GSLYSNK2DQZVCTNXMQQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.600511137 (1h2m48.381444525s) [noise=true , address=64.49.13.138:4483 , piece_id=E6KQ3CMVTHOZ4YFXGA4KERL6JMQFWLQXSDV77E26TROURWNEVSGA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.600544340 (1h2m48.381477722s) [noise=true , address=77.90.31.10:4533 , piece_id=CB2PNWRK4WGQD26LNCOQZYHM4PCHPKQZSV7GP7QSFZ4GDGBBK2FQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.600948113 (1h2m48.381881495s) [noise=true , address=79.127.205.246:4288 , piece_id=5FB3F4PGGRAWPFZJVOQVXOZ7DOGP4BX4A4725GFJGBYTI27EBT6A, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.602477077 (1h2m48.38341044s) [noise=true , address=156.146.43.225:4434 , piece_id=45W53IHFQDARAUDQYHPRTAVATANBAKXAH7F7MOVVZZH3DYUGQLSQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.714954727 (1h2m48.495888095s) [upload=971]: Next returned nil segment Jul 26 22:54:32.714976186 (1h2m48.495909574s) [upload=971]: Waiting for error group managing segments... Jul 26 22:54:32.734167577 (1h2m48.515100939s) [noise=true , address=77.90.31.11:4585 , piece_id=DQKVLVHJ4LOSIKDY73DVPDU2DNKCINTGZ4E7ZXN5YWE2LWSHMYMQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:32.750333100 (1h2m48.53126648s) [noise=true , address=77.90.31.21:4885 , piece_id=KEI7M7TMNBHZYEDH3IK2KUXCONNYTDZEXZIFOQ4HOYMV7E4LITTQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:33.093380297 (1h2m48.874313681s) [noise=true , address=64.49.13.138:4483 , piece_id=E6KQ3CMVTHOZ4YFXGA4KERL6JMQFWLQXSDV77E26TROURWNEVSGA, seg_pos={0 0}, upload=971]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:33.207266246 (1h2m48.988199618s) [noise=true , address=79.127.205.234:4605 , piece_id=E2GCBR5YSDUD4EZFLSF6GWVU6H7ZXWSQQTDQRO3C5HDCTOZ6FLFA, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:33.219489629 (1h2m49.000423012s) [noise=true , address=79.127.205.245:4244 , piece_id=EYN5ZWSWAXAKOGVRJHJXBESOPUG36M7SQXV6ZGB7LCLPDHU6JYNQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:33.220328517 (1h2m49.001261894s) [noise=true , address=79.127.205.246:4288 , piece_id=5FB3F4PGGRAWPFZJVOQVXOZ7DOGP4BX4A4725GFJGBYTI27EBT6A, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:33.701703946 (1h2m49.482637328s) [noise=true , address=79.127.201.215:4346 , piece_id=AFCOK2MOOS4ZTPLHWFICB5KHE7PZKMIXK2J3RAPSUUDLUSSO5RBQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:34.083112828 (1h2m49.86404622s) [noise=true , address=45.91.116.12:4229 , piece_id=HXNEO7TB6ZA2ALJAX25WTT4DRSQBKGDVRY57WNGZT6CB4LGB3CMA, seg_pos={0 0}, upload=971]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:35.969111312 (1h2m51.750044697s) [noise=true , address=77.90.31.10:4533 , piece_id=CB2PNWRK4WGQD26LNCOQZYHM4PCHPKQZSV7GP7QSFZ4GDGBBK2FQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:36.078077811 (1h2m51.859011176s) [noise=true , address=79.127.219.47:4048 , piece_id=UC37CIEOV35UCVM5MHFH23R5OFN47SLYKZMXRP66JHHNBE5D3OTA, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:36.328866515 (1h2m52.1097999s) [noise=true , address=64.49.13.140:4508 , piece_id=5PTUXJK6LL3GHBIJROFHT4RXJXBINNQC3M662JSKJLUTEQDCGWEA, seg_pos={0 0}, upload=971]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:36.777681573 (1h2m52.55861494s) [noise=true , address=156.146.43.225:4434 , piece_id=45W53IHFQDARAUDQYHPRTAVATANBAKXAH7F7MOVVZZH3DYUGQLSQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:36.877544518 (1h2m52.658477895s) [noise=true , address=77.90.31.13:4644 , piece_id=Y4D4T3SFZTF7NJJWPOMJLHOK23CVC7ZL3GSLYSNK2DQZVCTNXMQQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:37.912569159 (1h2m53.693502534s) [noise=true , address=79.127.220.97:4361 , piece_id=WQ3QJTHWC5ALCQZNGO7FB45HRRZW55XHKRTQRXHQFB7NJ7Y73F7A, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:41.263793146 (1h2m57.044726532s) [noise=true , address=45.91.116.9:4267 , piece_id=OE57ZC2HOABRYXKTQSFTOVQ7TQ7VHUSITVOMMFKGMPHSQ3XCRPHQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:51.099651867 (1h3m6.880585234s) [noise=true , address=45.91.116.14:4194 , piece_id=EQUU5VQL7BSI6IS5KK4JLLA6DRP4O6BJYVWMAHA24RH4DJNDAO3Q, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:55:04.514308623 (1h3m20.295242001s) [noise=true , address=45.91.116.8:4057 , piece_id=FCDXB7XKJYCZ3HFBISW5KLGGWN57S7XFBBPUHEQJEVNVVLY7CZ3A, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:55:27.566458463 (1h3m43.347391842s) [noise=true , address=79.127.201.215:4359 , piece_id=BZPNWTVQDFBF4QQM2JBT5RULPCHKCCKRDD365EVQZVMQJCIQIIDA, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:55:27.566510882 (1h3m43.347444265s) [seg_pos={0 0}, upload=971]: Segment reached optimal threshold of 14 pieces. Jul 26 22:55:27.566618049 (1h3m43.347551427s) [seg_pos={0 0}, upload=971]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 26 22:55:27.566754342 (1h3m43.347687726s) [upload=971]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:55:27.566760421 (1h3m43.347693803s) [upload=971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:55:27.625705218 (1h3m43.406638598s) [upload=971]: Done uploading segments... Jul 26 22:55:27.625741607 (1h3m43.406674975s) [upload=971]: Upload finished. err: <nil> Duration: 1m0.017062051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 191.082476ms FAIL (2025-07-26 22:55:27 UTC) orbiter warning: uplink: too many requests Duration: 191.11263ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X......X.............X................X................] duration: 853.035021ms FAIL (2025-07-26 22:55:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 853.123775ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..........X........X.X.X...XX.................X.X..........X] duration: 1m0.002779866s FAIL (2025-07-26 22:55:07 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002830858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX.X....X....X........X.....................................] duration: 385.854916ms FAIL (2025-07-26 22:54:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 385.900514ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 254.63483ms FAIL (2025-07-26 22:54:46 UTC) orbiter warning: uplink: too many requests Duration: 254.687599ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....X] duration: 1m0.033336885s FAIL (2025-07-26 22:54:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:32.813119492 (1h1m48.594052867s) [noise=true , address=45.91.116.14:4222 , piece_id=EGKSRBFK5WDNEMV3MLLJQHDYC4DLPBYO6B2Q4HUXQII4HCADSIIQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:32.915679332 (1h1m48.696612717s) [noise=true , address=64.49.13.170:4718 , piece_id=L7US2PVTOTPT4BN5EFUZF6M46LSMRAPZANBETGW2UU4EKR6F5N2A, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:12AcLsSmVz9sRtuDLs47WoWYC66YSbr2aXBjzBdjayg22daQ2Hp, address:64.49.13.170:4718): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:33.091484827 (1h1m48.872418207s) [noise=true , address=79.127.205.234:4617 , piece_id=U3VUFBGNHTM4QW534E4ZEAHUU2ALSAQH4WTNHFD7BFW4NPSQXUQA, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:33.103315699 (1h1m48.884249065s) [noise=true , address=79.127.205.233:4650 , piece_id=3MPBPF5OFZ2WNEYTFBK5XIXPPOKBRMAEH6NXX7NJFA2WK2VKW3HA, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:33.123562195 (1h1m48.904495565s) [noise=true , address=79.127.205.235:4678 , piece_id=NVTCFEKIYNNX5THIUP5F2T53QXMTJJ7UGX2SMNQE4L7FN7DYHT7Q, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:33.123610490 (1h1m48.904543872s) [noise=true , address=79.127.205.246:4269 , piece_id=PZN3EWSFH3T2FOL53ZR2BIHYAGNFWJO46BMBHVJVJ3TN4NBFSKWQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:33.179739024 (1h1m48.960672388s) [noise=true , address=45.91.116.9:4267 , piece_id=F2HYXNNHKK2TERXV6IMLGMV6WYOBM3PG6RZFXABUCPTB6ZU6PBNQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:33.302600826 (1h1m49.083534205s) [noise=true , address=79.127.219.51:4210 , piece_id=2VCMGRQ7Z3LFEDUXBZR6SGOEGWRRGD6WWFTGAE4OETO4FPD5ESYA, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:33.398568410 (1h1m49.179501787s) [noise=true , address=64.49.13.138:4482 , piece_id=OE3QVTYPBVNAZPWTLDEQD3TRGTRBLXXU3RUQYV5BEWPGJQG6AQUA, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:35.779566556 (1h1m51.560499922s) [noise=true , address=156.146.43.225:4434 , piece_id=KXLKD5TPW3UVUX6RQXJYK4ODSTEUS7UZUN2DMSFSQBCUAL6NQP4Q, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:39.616191602 (1h1m55.39712498s) [noise=true , address=64.49.13.155:4916 , piece_id=HIGJ36DKG5LEXQ4OWI3PIKXBQFKTZH4VFXPVXASFMB5QSWXWZBXQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:39.926132760 (1h1m55.707066125s) [noise=true , address=45.91.116.8:4065 , piece_id=5ZCTIVA4ZF4P4AQ6KG5MTBC7C66HIAWMCUUFKUQT2DBA55KNGH3A, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:17EPCQmYb816aegE9vPSoJYByMUEvhP4DUiabis2gh9QjXQC3s, address:45.91.116.8:4065): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:44.275335248 (1h2m0.056268618s) [noise=true , address=45.91.116.14:4206 , piece_id=ZAMVAXIDZ5WMTGWHUL7QS7EDM3MVRQT74NKG45BBIWSXIKGOAEDQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:05.973978655 (1h2m21.754912044s) [noise=true , address=77.90.31.10:4525 , piece_id=U2HFJAXWHGK4SBE7RT2DH4KE3QYHUCMLORQMTFHS5IA5YOJWYX4A, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:12.504493203 (1h2m28.285426588s) [noise=true , address=77.90.31.10:4533 , piece_id=LQ7JRQT3CXMOVK47Z2V5AX6IA3KNGB55PN6FMTOS24M5AMFNM2RQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:12.525194370 (1h2m28.30612776s) [noise=true , address=79.127.205.235:4686 , piece_id=QQ2Z2JLEYTGWFTR2KRQDIJRLMZVX2PBSD4UJPFVU5HT57GLCQNLQ, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.525655577 (1h2m28.306588966s) [noise=true , address=156.146.43.225:4434 , piece_id=4U6GSBRIV2BFML7JAFUFQ7DVCVTWBN4J7U3LPZFFNVX2OFEV6Y5A, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.525977780 (1h2m28.306911168s) [noise=true , address=77.90.31.13:4653 , piece_id=Z75RQ2ALZAPWI4LFDWJHQZLPTXA5JFG4LSPRAONIZXMOFXYODCRQ, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.526116442 (1h2m28.307049829s) [noise=true , address=79.127.205.246:4272 , piece_id=WDZMFL2I43UNK4KTLJOAAGAFYPR2WXIV6IMFGDTTQCJDH22LLDLQ, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.527131188 (1h2m28.308064575s) [noise=true , address=79.127.219.38:4010 , piece_id=NIKB2G6PVUVZNHDHDPD4STPVJZZBTEJLQOWZ32VEBEOW7CVD2UMA, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.527195999 (1h2m28.308129384s) [noise=true , address=79.127.220.97:4361 , piece_id=FVZ7OJACA2AXDXMU4NZTJNQBJV73H2LR3EQM42VEGNUAS4BSIEYA, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.528268214 (1h2m28.309201604s) [noise=true , address=64.49.13.138:4483 , piece_id=4CXJ3NQWDCW5YHOOBQDX423FB4LGZKAHCQM5ZAZMLIOAUTKWQGSQ, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.528347124 (1h2m28.309280511s) [noise=true , address=64.49.13.140:4508 , piece_id=RJJHI2AZHBL5DGAQDHIZRYHFFGFKTFOALU37JAH7DGGR3MTPTYDA, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.529864133 (1h2m28.31079752s) [noise=true , address=45.91.116.14:4194 , piece_id=3XFIUZE74HU4ACKIEWC5TMZYOG2456UTWWBZI5VOAV7Y5ROTRYLA, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.530303305 (1h2m28.311236696s) [noise=true , address=79.127.201.215:4353 , piece_id=SAZFEY3MHKXZEDBCXRSKOHOCN4L5KBGCGUEBXEMVOY7S2OIYAG7Q, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.565693539 (1h2m28.34662692s) [noise=true , address=77.90.31.13:4653 , piece_id=Z75RQ2ALZAPWI4LFDWJHQZLPTXA5JFG4LSPRAONIZXMOFXYODCRQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, address:77.90.31.13:4653): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:12.583435196 (1h2m28.364368579s) [noise=true , address=79.127.219.38:4010 , piece_id=NIKB2G6PVUVZNHDHDPD4STPVJZZBTEJLQOWZ32VEBEOW7CVD2UMA, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:12.635341566 (1h2m28.416274935s) [noise=true , address=156.146.43.225:4434 , piece_id=4U6GSBRIV2BFML7JAFUFQ7DVCVTWBN4J7U3LPZFFNVX2OFEV6Y5A, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: expected piece hash; used serial already exists in store; EOF Jul 26 22:54:12.730251091 (1h2m28.511184471s) [noise=true , address=45.91.116.14:4194 , piece_id=3XFIUZE74HU4ACKIEWC5TMZYOG2456UTWWBZI5VOAV7Y5ROTRYLA, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:13.115781374 (1h2m28.896714739s) [noise=true , address=79.127.205.246:4272 , piece_id=WDZMFL2I43UNK4KTLJOAAGAFYPR2WXIV6IMFGDTTQCJDH22LLDLQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:13.121922344 (1h2m28.90285572s) [noise=true , address=79.127.205.235:4686 , piece_id=QQ2Z2JLEYTGWFTR2KRQDIJRLMZVX2PBSD4UJPFVU5HT57GLCQNLQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:13.159472054 (1h2m28.94040542s) [noise=true , address=79.127.220.97:4361 , piece_id=FVZ7OJACA2AXDXMU4NZTJNQBJV73H2LR3EQM42VEGNUAS4BSIEYA, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:13.191993414 (1h2m28.972926801s) [noise=true , address=64.49.13.140:4508 , piece_id=RJJHI2AZHBL5DGAQDHIZRYHFFGFKTFOALU37JAH7DGGR3MTPTYDA, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:16.118043660 (1h2m31.898977036s) [noise=true , address=64.49.13.138:4483 , piece_id=4CXJ3NQWDCW5YHOOBQDX423FB4LGZKAHCQM5ZAZMLIOAUTKWQGSQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:32.408607043 (1h2m48.189540424s) [noise=true , address=79.127.201.215:4353 , piece_id=SAZFEY3MHKXZEDBCXRSKOHOCN4L5KBGCGUEBXEMVOY7S2OIYAG7Q, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1227Uk4K8hmXrae9npcgjziCNhwfdfdDPdgEryAy5beiFaSmd1W, address:79.127.201.215:4353): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:54:32.408658110 (1h2m48.189591498s) [seg_pos={0 0}, upload=955]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:54:32.438371519 (1h2m48.219304899s) [upload=955]: Done uploading segments... Jul 26 22:54:32.438402298 (1h2m48.219335675s) [upload=955]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.033398337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X..X......X.............X................X...............] duration: 429.058131ms FAIL (2025-07-26 22:54:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.096805ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 406.533933ms FAIL (2025-07-26 22:53:59 UTC) orbiter warning: uplink: too many requests Duration: 406.563498ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX.X....X....X........X....................................] duration: 735.398417ms FAIL (2025-07-26 22:53:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 735.428693ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........X........X.X.X...XX.................X.X..........] duration: 14.101728331s FAIL (2025-07-26 22:53:34 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.101767008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....] duration: 30.790620249s FAIL (2025-07-26 22:53:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== pb.BatchRequestItem_SegmentBegin Jul 26 22:49:01.022497827 (57m16.803431216s) [seg_pos={0 0}, upload=881]: Begin upload segment... Jul 26 22:49:01.022580489 (57m16.803513879s) [seg_pos={0 0}, upload=881]: Done begin upload segment. Jul 26 22:49:01.022617556 (57m16.803550944s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023291722 (57m16.804225102s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023899131 (57m16.804832515s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024446671 (57m16.805380054s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024966954 (57m16.805900344s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.025508416 (57m16.806441807s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026008754 (57m16.806942141s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026427269 (57m16.80736065s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026853310 (57m16.807786706s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027286217 (57m16.808219607s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027718535 (57m16.808651921s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.029646196 (57m16.810579586s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030101602 (57m16.811034994s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030548215 (57m16.811481598s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030961321 (57m16.811894709s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031462001 (57m16.812395386s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031844379 (57m16.812777746s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.032259287 (57m16.813192676s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.102276736 (57m16.883210122s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.102735014 (57m16.883668406s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.123020337 (57m16.903953719s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170593572 (57m16.951526952s) [upload=881]: Next returned nil segment Jul 26 22:49:01.170610860 (57m16.951544242s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170633877 (57m16.951567243s) [upload=881]: Waiting for error group managing segments... Jul 26 22:49:01.195133094 (57m16.976066474s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.202867381 (57m16.98380075s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.226760035 (57m17.007693403s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.229442519 (57m17.010375908s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.715130515 (57m17.496063893s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.728479686 (57m17.509413058s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.196712183 (57m20.97764556s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.652598133 (57m21.433531515s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:06.470679063 (57m22.25161244s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:07.317498912 (57m23.098432629s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.623872606 (57m24.404805973s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1mY7sRFAeDLmKPqq8BgwnGtkdPTsDRR6soTBshsn4qhP7evARb, address:64.49.13.140:4515): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.736598453 (57m24.51753183s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:52.335446351 (58m8.116379721s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:50:00.909179814 (58m16.690113181s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:50:00.909239469 (58m16.690172853s) [seg_pos={0 0}, upload=881]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:50:00.935905580 (58m16.716838959s) [upload=881]: Done uploading segments... Jul 26 22:50:00.935961200 (58m16.716894583s) [upload=881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 30.790658883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX..X..X......X.............X................X..............] duration: 616.19925ms FAIL (2025-07-26 22:53:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 616.250881ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 194.542368ms FAIL (2025-07-26 22:52:58 UTC) orbiter warning: uplink: too many requests Duration: 194.586681ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX.X....X....X........X...................................] duration: 474.93512ms FAIL (2025-07-26 22:52:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 474.974744ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..XX.X....X....X........X..................................] duration: 580.006888ms FAIL (2025-07-26 22:52:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 580.043055ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [............X........X.X.X...XX.................X.X.........] duration: 8.641585244s FAIL (2025-07-26 22:52:19 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.641604382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX...] duration: 12.4304335s FAIL (2025-07-26 22:52:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== pb.BatchRequestItem_SegmentBegin Jul 26 22:49:01.022497827 (57m16.803431216s) [seg_pos={0 0}, upload=881]: Begin upload segment... Jul 26 22:49:01.022580489 (57m16.803513879s) [seg_pos={0 0}, upload=881]: Done begin upload segment. Jul 26 22:49:01.022617556 (57m16.803550944s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023291722 (57m16.804225102s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023899131 (57m16.804832515s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024446671 (57m16.805380054s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024966954 (57m16.805900344s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.025508416 (57m16.806441807s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026008754 (57m16.806942141s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026427269 (57m16.80736065s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026853310 (57m16.807786706s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027286217 (57m16.808219607s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027718535 (57m16.808651921s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.029646196 (57m16.810579586s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030101602 (57m16.811034994s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030548215 (57m16.811481598s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030961321 (57m16.811894709s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031462001 (57m16.812395386s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031844379 (57m16.812777746s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.032259287 (57m16.813192676s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.102276736 (57m16.883210122s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.102735014 (57m16.883668406s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.123020337 (57m16.903953719s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170593572 (57m16.951526952s) [upload=881]: Next returned nil segment Jul 26 22:49:01.170610860 (57m16.951544242s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170633877 (57m16.951567243s) [upload=881]: Waiting for error group managing segments... Jul 26 22:49:01.195133094 (57m16.976066474s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.202867381 (57m16.98380075s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.226760035 (57m17.007693403s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.229442519 (57m17.010375908s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.715130515 (57m17.496063893s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.728479686 (57m17.509413058s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.196712183 (57m20.97764556s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.652598133 (57m21.433531515s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:06.470679063 (57m22.25161244s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:07.317498912 (57m23.098432629s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.623872606 (57m24.404805973s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1mY7sRFAeDLmKPqq8BgwnGtkdPTsDRR6soTBshsn4qhP7evARb, address:64.49.13.140:4515): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.736598453 (57m24.51753183s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:52.335446351 (58m8.116379721s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:50:00.909179814 (58m16.690113181s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:50:00.909239469 (58m16.690172853s) [seg_pos={0 0}, upload=881]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:50:00.935905580 (58m16.716838959s) [upload=881]: Done uploading segments... Jul 26 22:50:00.935961200 (58m16.716894583s) [upload=881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 12.43047775s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)