Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........................................X........X.X.X...XX.] duration: 4.987162153s FAIL (2025-07-26 22:31:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.987183675s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........................................X........X.X.X...XX] duration: 1m0.00063588s FAIL (2025-07-26 22:31:15 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000652214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [............XX..XXX..X..XX..X..XX.X....X....X........X......] duration: 1.114872713s FAIL (2025-07-26 22:31:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.114906933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 231.383667ms FAIL (2025-07-26 22:31:00 UTC) orbiter warning: uplink: too many requests Duration: 231.420682ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X..X........XXX.XXX.X.XX..X..X......X.............X....] duration: 452.889304ms FAIL (2025-07-26 22:30:51 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 452.954895ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 195.576028ms FAIL (2025-07-26 22:30:37 UTC) orbiter warning: uplink: too many requests Duration: 195.607825ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.84352625s FAIL (2025-07-26 22:30:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.843580658s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X..X........XXX.XXX.X.XX..X..X......X.............X...] duration: 450.840526ms FAIL (2025-07-26 22:30:31 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 450.882768ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............XX..XXX..X..XX..X..XX.X....X....X........X.....] duration: 562.993517ms FAIL (2025-07-26 22:30:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 563.175298ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......X..X........XXX.XXX.X.XX..X..X......X.............X..] duration: 515.983574ms FAIL (2025-07-26 22:30:16 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 516.038922ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................................X........X.X.X...X] duration: 1m0.001177122s FAIL (2025-07-26 22:30:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001276067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration: 1m0.0392517s FAIL (2025-07-26 22:30:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== DRNV6QVZDYWTCCCEC7CNHCY3Z7YVMWEK3K2M4WSA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.872330524 (37m24.653263907s) [noise=true , address=64.49.13.170:4721 , piece_id=U4VAMA23OUVSBMM2ZWUH3RBRWV4R6IUAVOTMG57S7LKA4CLLDFYA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.872852006 (37m24.653785395s) [noise=true , address=104.36.82.41:4779 , piece_id=TWCQUEWFOQJP2HASYDMNVBO3A4Q7TUKCEPX346KFH3U6OHWYRJSA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.872977116 (37m24.653910502s) [noise=true , address=79.127.205.235:4692 , piece_id=GKFKK4SFO6KDDVXQMKRI6VUVKBWIGWRLLMRLZDYTKIIQJ3KF3KYQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873348563 (37m24.654281945s) [noise=true , address=45.91.116.13:4175 , piece_id=KZQ25XSWEXYLXML66QVZLYYVO54FEYAZYCVFZNX27N3FHOFXIW4A, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873460795 (37m24.654394183s) [noise=true , address=104.36.82.42:4813 , piece_id=4LHK3DNMMJLS4WEUTJ5X6BDLXXUAM2VLWZFKXQMASJYLKY7TZCWQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873814983 (37m24.654748371s) [noise=true , address=104.36.82.45:4928 , piece_id=4VLYUSNGBNO2SMUSC2VYAHYCDWEU44F4FDWNUYMW6VDXKDHZ3UEQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873904265 (37m24.654837652s) [noise=true , address=79.127.201.215:4354 , piece_id=SJKK5A7R4X5B4UBQDIRIKB65X2EFX7UAWGXWSFTVKZOI4NCSJUBA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874272710 (37m24.655206097s) [noise=true , address=79.127.219.47:4048 , piece_id=SUE6QBJAZUSUGVDDBQC35ZU66K57MHRHZIIYESFME5SXLMUH7G4A, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874350404 (37m24.655283792s) [noise=true , address=104.36.82.52:4088 , piece_id=4HYQBC7HQQXUDYARJAQYVZI34VLXFO7MUNI52YN2DFI4BHT6IGEQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874729933 (37m24.655663317s) [noise=true , address=79.127.205.233:4652 , piece_id=BBD5SHWY6D2LOZOCSG734PYP3BC32EKRVG4WRKVG65L5FIKVBVBQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874798536 (37m24.65573192s) [noise=true , address=104.36.82.11:4143 , piece_id=K4EPYJLKKIRGS565OPQO5CSP7W6DLSQKPG3O2JLIAMKBLDMKCOAA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875192974 (37m24.656126364s) [noise=true , address=104.36.82.12:4160 , piece_id=K6XAPHAZWQK4EZE4AZFOI5LUWRO4PDSCCRLTUQM3BDV4RZ2IFRXA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875280310 (37m24.656213703s) [noise=true , address=104.36.82.36:4651 , piece_id=NSLFJ3RIELJLG7TJXBB7X4NHZDCJ2J33ODTATJRNG3B4OPYAY62Q, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875649341 (37m24.656582728s) [noise=true , address=45.91.116.14:4206 , piece_id=WKRPJDJC4GYUAX6GCV27HZABRW6CG2XPFFFMLL6ZDMCOVN2SRQIA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875758061 (37m24.656691447s) [noise=true , address=104.36.82.40:4942 , piece_id=KRPCZBHIU5ZBUV2PQZBRQ3ZJXSZZIPSNFRRLN3L65SYWDG4X5U4Q, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.876344348 (37m24.657277726s) [noise=true , address=79.127.205.245:4220 , piece_id=UW3UIJ6MPYAGZXSI7MOCF2ODRBD52VKSDG7GPKFYFZ7VH4KYTO2A, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:09.139975922 (37m24.920909285s) [upload=582]: Next returned nil segment Jul 26 22:29:09.140138981 (37m24.921072343s) [upload=582]: Waiting for error group managing segments... Jul 26 22:29:09.474617798 (37m25.255551165s) [noise=true , address=79.127.205.233:4652 , piece_id=BBD5SHWY6D2LOZOCSG734PYP3BC32EKRVG4WRKVG65L5FIKVBVBQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:09.476801572 (37m25.25773494s) [noise=true , address=79.127.205.235:4692 , piece_id=GKFKK4SFO6KDDVXQMKRI6VUVKBWIGWRLLMRLZDYTKIIQJ3KF3KYQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:09.561523995 (37m25.342457365s) [noise=true , address=79.127.205.245:4220 , piece_id=UW3UIJ6MPYAGZXSI7MOCF2ODRBD52VKSDG7GPKFYFZ7VH4KYTO2A, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:11.507316685 (37m27.288250074s) [noise=true , address=79.127.219.47:4048 , piece_id=SUE6QBJAZUSUGVDDBQC35ZU66K57MHRHZIIYESFME5SXLMUH7G4A, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:11.941822852 (37m27.722756237s) [noise=true , address=64.49.13.170:4721 , piece_id=U4VAMA23OUVSBMM2ZWUH3RBRWV4R6IUAVOTMG57S7LKA4CLLDFYA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:29:12.834296579 (37m28.615229957s) [noise=true , address=79.127.201.215:4354 , piece_id=SJKK5A7R4X5B4UBQDIRIKB65X2EFX7UAWGXWSFTVKZOI4NCSJUBA, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:13.872012428 (37m29.652945807s) [noise=true , address=45.91.116.10:4100 , piece_id=VPHKMJGRLKOKDRNV6QVZDYWTCCCEC7CNHCY3Z7YVMWEK3K2M4WSA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: storage node overloaded, request limit: 2000 Jul 26 22:29:19.175006841 (37m34.955940211s) [noise=true , address=45.91.116.14:4206 , piece_id=WKRPJDJC4GYUAX6GCV27HZABRW6CG2XPFFFMLL6ZDMCOVN2SRQIA, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:21.249247070 (37m37.030180453s) [noise=true , address=45.91.116.13:4175 , piece_id=KZQ25XSWEXYLXML66QVZLYYVO54FEYAZYCVFZNX27N3FHOFXIW4A, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:04.345974914 (38m20.126908287s) [noise=true , address=104.36.82.52:4088 , piece_id=4HYQBC7HQQXUDYARJAQYVZI34VLXFO7MUNI52YN2DFI4BHT6IGEQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:30:08.725951783 (38m24.506885162s) [noise=true , address=104.36.82.40:4942 , piece_id=KRPCZBHIU5ZBUV2PQZBRQ3ZJXSZZIPSNFRRLN3L65SYWDG4X5U4Q, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:104.36.82.40:4942): protocol: rpc: dial tcp 104.36.82.40:4942: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726063777 (38m24.506997155s) [noise=true , address=104.36.82.41:4779 , piece_id=TWCQUEWFOQJP2HASYDMNVBO3A4Q7TUKCEPX346KFH3U6OHWYRJSA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1eRovJAp3AyUHP4qkuDvSSGYb9TG6U4tSNzniGJBeYLhkmiUVy, address:104.36.82.41:4779): protocol: rpc: dial tcp 104.36.82.41:4779: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726143819 (38m24.507077204s) [noise=true , address=104.36.82.11:4143 , piece_id=K4EPYJLKKIRGS565OPQO5CSP7W6DLSQKPG3O2JLIAMKBLDMKCOAA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12ipThd6PwZUpCXy3ML4qqMd9dmRLmxKjVNCNoT5wCg1szdWp8U, address:104.36.82.11:4143): protocol: rpc: dial tcp 104.36.82.11:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726212930 (38m24.507146312s) [noise=true , address=104.36.82.12:4160 , piece_id=K6XAPHAZWQK4EZE4AZFOI5LUWRO4PDSCCRLTUQM3BDV4RZ2IFRXA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12dggZZRv5coYBNJEBvHNqWtMSqYkZoXYmitc9nBKJZftnsX17D, address:104.36.82.12:4160): protocol: rpc: dial tcp 104.36.82.12:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726277487 (38m24.507210863s) [noise=true , address=104.36.82.56:4233 , piece_id=56UZRSJOWFMOUEGYU4CFLN4O5I4HHJ7JRPEULXON2XYAGROXDZPQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726339540 (38m24.507272924s) [noise=true , address=104.36.82.42:4813 , piece_id=4LHK3DNMMJLS4WEUTJ5X6BDLXXUAM2VLWZFKXQMASJYLKY7TZCWQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1Egk1BCBmtjUZcwJKpoQjfrubA2czJKMEai5bzDqkPmXTRhaB6, address:104.36.82.42:4813): protocol: rpc: dial tcp 104.36.82.42:4813: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726402211 (38m24.507335589s) [noise=true , address=104.36.82.45:4928 , piece_id=4VLYUSNGBNO2SMUSC2VYAHYCDWEU44F4FDWNUYMW6VDXKDHZ3UEQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12B6ymfqNvqiDsAbkF8H4ExxEuYaYb7gYQ45Y5GF4PGAAwmjbyd, address:104.36.82.45:4928): protocol: rpc: dial tcp 104.36.82.45:4928: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726467024 (38m24.50740041s) [noise=true , address=104.36.82.36:4651 , piece_id=NSLFJ3RIELJLG7TJXBB7X4NHZDCJ2J33ODTATJRNG3B4OPYAY62Q, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12tD1Zfi4ZxujF9pVdgW5k2wnejXsa7tf1u25FuucRXeneGP3dz, address:104.36.82.36:4651): protocol: write tcp 159.223.143.39:43616->104.36.82.36:4651: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726489768 (38m24.507423159s) [seg_pos={0 0}, upload=582]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:30:08.761600623 (38m24.542534006s) [upload=582]: Done uploading segments... Jul 26 22:30:08.761635235 (38m24.542568618s) [upload=582]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.039269014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [........X..X........XXX.XXX.X.XX..X..X......X.............X.] duration: 2.001677008s FAIL (2025-07-26 22:29:48 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 2.001717878s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 515.273029ms FAIL (2025-07-26 22:29:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 515.306952ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 221.064152ms FAIL (2025-07-26 22:29:41 UTC) orbiter warning: uplink: too many requests Duration: 221.097603ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 220.865789ms FAIL (2025-07-26 22:29:24 UTC) orbiter warning: uplink: too many requests Duration: 220.900138ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..............XX..XXX..X..XX..X..XX.X....X....X........X....] duration: 456.197031ms FAIL (2025-07-26 22:29:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.232604ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 611.778316ms FAIL (2025-07-26 22:29:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 611.807268ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........X..X........XXX.XXX.X.XX..X..X......X.............X] duration: 405.750901ms FAIL (2025-07-26 22:29:12 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 405.782476ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX] duration: 1m0.03076395s FAIL (2025-07-26 22:29:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== eg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.781832705 (36m24.562766094s) [noise=true , address=79.127.201.215:4355 , piece_id=E2A5I7QPSE4BJ3GTRKVBMAZQ5WQLQOPDH4FWN5ZEIQU47AAZHM4A, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.782330540 (36m24.563264048s) [noise=true , address=79.127.205.245:4236 , piece_id=6BC2XEUIHGPXXN57ABQQU5HRACK2A6NQP7FNHI3XZ7LOB2WCXRFQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.782464497 (36m24.563397886s) [noise=true , address=79.127.205.246:4287 , piece_id=CCVRY4D4VQT5NPPTGPZHVSHFI3KQFC7I7L73VK56ZA35O3DL5QQQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.782878492 (36m24.563811878s) [noise=true , address=104.36.82.18:4393 , piece_id=CD2FO2WJDQPGBRV6XL2RVYFHQSD46LFFAAWBEQCTK4YT2GYINARA, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783111638 (36m24.564045026s) [noise=true , address=104.36.82.40:4967 , piece_id=APXWWFDCX3R22UNOS6DU7TQZDOESBJU7IZAE7XK33VIKPDACJKGQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783530994 (36m24.56446438s) [noise=true , address=79.127.219.38:4007 , piece_id=REDAVJSLA3XYRC6ADZ4JLUXE26GXTWBNMRPEWPYZXJ7VU4VVOH5A, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783765643 (36m24.56469903s) [noise=true , address=104.36.82.44:4890 , piece_id=ZR2KTG6XPPKEUTKUPFC5C2LGEPNMYVBEMILEQTYXEJB3DY3CNUDQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783946386 (36m24.564879778s) [noise=true , address=104.36.82.44:4897 , piece_id=FPU2DV6FSRXHCQZIFDWYWRZCGMMML5HLP22SET4M3XLVBO37E53Q, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.784412344 (36m24.565345728s) [noise=true , address=64.49.13.138:4483 , piece_id=T43HVJMVKJMD6WAA5QHO4URWNAWJNBP6FS7FPQYS6EKPGDC3KFVQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.784460953 (36m24.565394339s) [noise=true , address=104.36.82.56:4239 , piece_id=TTAN677QIV3CUW432OBTIX3DY7WLX7GNDD25OS6FMA5UYQMZGKGQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.784942001 (36m24.565875392s) [noise=true , address=77.90.31.18:4832 , piece_id=742YMK223EYD7JFFCB2TQBOQ6NG3OHT2E24MWO6XIJN2DIVTKU3A, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786106520 (36m24.56703991s) [noise=true , address=104.36.82.7:4916 , piece_id=SRQG2L36L2I6OYFRMDUXVZNS3UMUKORVIASNTAWBPHYHKNA37JMA, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786393751 (36m24.567327141s) [noise=true , address=104.36.82.10:4089 , piece_id=GFG6P4EYVFVWPMITY6OKTR5L2ICM6UZNBNPJUQL6TSED34LHNMXQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786632451 (36m24.567565839s) [noise=true , address=79.127.219.48:4090 , piece_id=JPFMYBZTM2RW7HYKLUCTCM6UA6SUJDBCCA46J2WS3Q6BYNHMI7FQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786826948 (36m24.567760337s) [noise=true , address=104.36.82.11:4125 , piece_id=IF5726HY35L7Q47QPCK7WMOJSBVFVJ7XQASG3BQZQMX4BJYEUYOQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.787078509 (36m24.568011888s) [noise=true , address=77.90.31.15:4726 , piece_id=JJ7Y6JR7BULYCKJVJPHAJXURC4AQDP3EOZQYIQNJSMHW3CB3GYMQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.869617862 (36m24.650551257s) [upload=568]: Next returned nil segment Jul 26 22:28:08.869639727 (36m24.650573114s) [upload=568]: Waiting for error group managing segments... Jul 26 22:28:08.903844258 (36m24.684777623s) [noise=true , address=77.90.31.18:4832 , piece_id=742YMK223EYD7JFFCB2TQBOQ6NG3OHT2E24MWO6XIJN2DIVTKU3A, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:08.997481088 (36m24.778414463s) [noise=true , address=79.127.219.38:4007 , piece_id=REDAVJSLA3XYRC6ADZ4JLUXE26GXTWBNMRPEWPYZXJ7VU4VVOH5A, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.125047128 (36m24.905980512s) [noise=true , address=79.127.219.48:4090 , piece_id=JPFMYBZTM2RW7HYKLUCTCM6UA6SUJDBCCA46J2WS3Q6BYNHMI7FQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.283982565 (36m25.064915935s) [noise=true , address=77.90.31.15:4726 , piece_id=JJ7Y6JR7BULYCKJVJPHAJXURC4AQDP3EOZQYIQNJSMHW3CB3GYMQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.389541409 (36m25.170474777s) [noise=true , address=79.127.205.245:4236 , piece_id=6BC2XEUIHGPXXN57ABQQU5HRACK2A6NQP7FNHI3XZ7LOB2WCXRFQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.477572185 (36m25.258505554s) [noise=true , address=79.127.205.246:4287 , piece_id=CCVRY4D4VQT5NPPTGPZHVSHFI3KQFC7I7L73VK56ZA35O3DL5QQQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:14.011496993 (36m29.792430379s) [noise=true , address=64.49.13.138:4483 , piece_id=T43HVJMVKJMD6WAA5QHO4URWNAWJNBP6FS7FPQYS6EKPGDC3KFVQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:22.943945444 (36m38.724878812s) [noise=true , address=79.127.201.215:4355 , piece_id=E2A5I7QPSE4BJ3GTRKVBMAZQ5WQLQOPDH4FWN5ZEIQU47AAZHM4A, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:29:08.617901143 (37m24.39883452s) [noise=true , address=104.36.82.18:4393 , piece_id=CD2FO2WJDQPGBRV6XL2RVYFHQSD46LFFAAWBEQCTK4YT2GYINARA, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12fDTUq74fpQVaFNsfxPmKwMpG9eUfFDcjPd5FutuqXiBp9R5EN, address:104.36.82.18:4393): protocol: rpc: dial tcp 104.36.82.18:4393: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.617992630 (37m24.398925992s) [noise=true , address=104.36.82.7:4916 , piece_id=SRQG2L36L2I6OYFRMDUXVZNS3UMUKORVIASNTAWBPHYHKNA37JMA, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12fifEPABGWmVzxBHNzhjh3r4ue3RRSeNdydifpXruXtH4FELBK, address:104.36.82.7:4916): protocol: rpc: dial tcp 104.36.82.7:4916: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618231339 (37m24.399164717s) [noise=true , address=104.36.82.11:4125 , piece_id=IF5726HY35L7Q47QPCK7WMOJSBVFVJ7XQASG3BQZQMX4BJYEUYOQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:4125): protocol: rpc: dial tcp 104.36.82.11:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618318480 (37m24.399251863s) [noise=true , address=104.36.82.44:4897 , piece_id=FPU2DV6FSRXHCQZIFDWYWRZCGMMML5HLP22SET4M3XLVBO37E53Q, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU, address:104.36.82.44:4897): protocol: rpc: dial tcp 104.36.82.44:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618394683 (37m24.39932806s) [noise=true , address=104.36.82.10:4089 , piece_id=GFG6P4EYVFVWPMITY6OKTR5L2ICM6UZNBNPJUQL6TSED34LHNMXQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 159.223.143.39:34384->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618462310 (37m24.399395687s) [noise=true , address=104.36.82.44:4890 , piece_id=ZR2KTG6XPPKEUTKUPFC5C2LGEPNMYVBEMILEQTYXEJB3DY3CNUDQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618471039 (37m24.399404419s) [noise=true , address=104.36.82.52:4094 , piece_id=YFYPN3FRCIHUN22VGHWTK53JTQKDGPPVQVAWQGYORBLQWDKN7XQA, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: rpc: dial tcp 104.36.82.52:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618560947 (37m24.399494325s) [noise=true , address=104.36.82.56:4239 , piece_id=TTAN677QIV3CUW432OBTIX3DY7WLX7GNDD25OS6FMA5UYQMZGKGQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: rpc: dial tcp 104.36.82.56:4239: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618599925 (37m24.399533288s) [noise=true , address=104.36.82.12:4152 , piece_id=J4Y7ZDRMQTXU6IVCFEC7U4JV23HQA47CYVQONKP62ASLXN6JYFVQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:104.36.82.12:4152): protocol: write tcp 159.223.143.39:58020->104.36.82.12:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618618191 (37m24.399551572s) [noise=true , address=104.36.82.40:4967 , piece_id=APXWWFDCX3R22UNOS6DU7TQZDOESBJU7IZAE7XK33VIKPDACJKGQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: write tcp 159.223.143.39:35046->104.36.82.40:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618677546 (37m24.399610934s) [seg_pos={0 0}, upload=568]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:29:08.644975149 (37m24.425908526s) [upload=568]: Done uploading segments... Jul 26 22:29:08.645009948 (37m24.425943394s) [upload=568]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.030874016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 516.687863ms FAIL (2025-07-26 22:28:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 516.70933ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...............XX..XXX..X..XX..X..XX.X....X....X........X...] duration: 453.101199ms FAIL (2025-07-26 22:28:45 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.134146ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 51.546106ms FAIL (2025-07-26 22:28:45 UTC) orbiter warning: uplink: too many requests Duration: 51.572545ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X........X.X.X...] duration: 4.119879275s FAIL (2025-07-26 22:28:29 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.119902276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 658.894921ms FAIL (2025-07-26 22:28:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 659.010631ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXX] duration: 1m0.039342136s FAIL (2025-07-26 22:28:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== =555]: Got next segment. Inline: false Jul 26 22:27:08.521554810 (35m24.302488195s) [upload=555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:27:08.521561305 (35m24.302494689s) [upload=555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:27:08.566725134 (35m24.34765863s) [seg_pos={0 0}, upload=555]: Begin upload segment... Jul 26 22:27:08.566868456 (35m24.347801846s) [seg_pos={0 0}, upload=555]: Done begin upload segment. Jul 26 22:27:08.566926374 (35m24.347859781s) [noise=true , address=79.127.201.215:4358 , piece_id=2ZGKHBGSRVFZAXKJEN3GBZCEJVPEIHXZ2ZIAARFJRJRSQQBK5XDA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.566948436 (35m24.347881825s) [noise=true , address=77.90.31.16:4746 , piece_id=5DBORRAQIHTBNXJYJEOOBQC4YOCLPWCG5H3FGYCZAZ5E7BTUXZJQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.567073386 (35m24.34800677s) [noise=true , address=45.91.116.5:4310 , piece_id=D4PIYEMA7PFEUESZ7BPNOHPEONWQKBRL2BGWIHG366PFV77MPGOA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.567519381 (35m24.348452772s) [noise=true , address=64.49.13.138:4483 , piece_id=5WKBRLGZZV43P3AMSSUVRWWIYSF54MYZZ33AVQVV463WLYBGP26A, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.567626007 (35m24.34855939s) [noise=true , address=104.36.82.52:4090 , piece_id=DC6RDKRDWHE3GI7VCMOGHV5YH5ACOJ5UQRWNXJ46IVYEBSLPHG5Q, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.567987293 (35m24.348920672s) [noise=true , address=79.127.205.233:4639 , piece_id=EBVCFHT5G2LHKS2W6WTQAKPXXS3PEESY22MJFB736KG6LHCQY7VQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.568455886 (35m24.349389271s) [noise=true , address=79.127.205.246:4285 , piece_id=RB4TDSIXLN4VAQEXQEARJFDF24MO4EOMB756URVLHOZ3CRAAJQBQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.568561311 (35m24.3494947s) [noise=true , address=104.36.82.51:4081 , piece_id=Y3UHTSYTIWFWRSQFFXTJYD3SI4DFCIB3QUWSEQHQAL7N3Y2KKSCQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.568996915 (35m24.349930339s) [noise=true , address=104.36.82.4:4827 , piece_id=AOIJVARJ5LRFWLGTP24ZBT2MGQTER2JKQIH4XDO6MKPH4I2OUFAQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.569015289 (35m24.349948675s) [noise=true , address=104.36.82.16:4300 , piece_id=SH6UOAO5HJIBRNMA34PQ2YAXYLBVLTMGQ6ULB6IHKP3F4NAYWS7A, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.569455644 (35m24.350389031s) [noise=true , address=79.127.205.235:4689 , piece_id=HNHDF4CU5R6BGSVI3KWNUHGPNXE25U3GVYZGUXUDCES7CP4OZEOA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.569872680 (35m24.350806068s) [noise=true , address=104.36.82.21:4499 , piece_id=PRNLGOYXS3BQJEJAXAO6QJD4O4LQUFQYKZQEJDGKVBF742HJAESA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.569989884 (35m24.35092328s) [noise=true , address=45.91.116.14:4222 , piece_id=4YKN5ZIM2N2FV53GO3H7S6JHGIMSN5KDFV3VKHIHCXRMAKILH67A, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.570344970 (35m24.351278357s) [noise=true , address=79.127.205.234:4615 , piece_id=CDSJAQOFK7HSEQMST27L4YBTCSOJETH6MDRXT46WQSSRWI3RQLXA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.570820355 (35m24.351753743s) [noise=true , address=79.127.205.245:4243 , piece_id=R5ZNLJJP5JQCWIQ2DLO3IDWJI6GBY3WASG7HRLNTDIXZAFPPJACQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.575996870 (35m24.356930245s) [noise=true , address=104.36.82.56:4234 , piece_id=6E6KBP5GCJHYZIAFEP5J4RWR7TJ5SV5AFSA6FRWLNUSAYAVEYMTQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.578658024 (35m24.359591411s) [noise=true , address=79.127.219.38:4015 , piece_id=WZD2VEADTQZSJYWWQ665VCZNU3CWL3Z5RY2PUSA56MLYNOGZJYLA, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.587960422 (35m24.368893809s) [noise=true , address=45.91.116.11:4146 , piece_id=AQPQUYI3YCZC23ALHTLJYZ4YGWZ6XRQMS3PH62F63EV7YASALXHQ, seg_pos={0 0}, upload=555]: Uploading piece... Jul 26 22:27:08.683348272 (35m24.46428164s) [upload=555]: Next returned nil segment Jul 26 22:27:08.683365120 (35m24.464298501s) [upload=555]: Waiting for error group managing segments... Jul 26 22:27:08.750506310 (35m24.531439688s) [noise=true , address=77.90.31.16:4746 , piece_id=5DBORRAQIHTBNXJYJEOOBQC4YOCLPWCG5H3FGYCZAZ5E7BTUXZJQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:08.784020342 (35m24.564953712s) [noise=true , address=79.127.219.38:4015 , piece_id=WZD2VEADTQZSJYWWQ665VCZNU3CWL3Z5RY2PUSA56MLYNOGZJYLA, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:09.151235321 (35m24.932168691s) [noise=true , address=79.127.205.235:4689 , piece_id=HNHDF4CU5R6BGSVI3KWNUHGPNXE25U3GVYZGUXUDCES7CP4OZEOA, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:09.169744980 (35m24.950678355s) [noise=true , address=79.127.205.245:4243 , piece_id=R5ZNLJJP5JQCWIQ2DLO3IDWJI6GBY3WASG7HRLNTDIXZAFPPJACQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:09.202860077 (35m24.983793443s) [noise=true , address=79.127.205.234:4615 , piece_id=CDSJAQOFK7HSEQMST27L4YBTCSOJETH6MDRXT46WQSSRWI3RQLXA, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:09.211433807 (35m24.992367491s) [noise=true , address=79.127.205.233:4639 , piece_id=EBVCFHT5G2LHKS2W6WTQAKPXXS3PEESY22MJFB736KG6LHCQY7VQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:09.362863020 (35m25.143796387s) [noise=true , address=79.127.205.246:4285 , piece_id=RB4TDSIXLN4VAQEXQEARJFDF24MO4EOMB756URVLHOZ3CRAAJQBQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:12.882201179 (35m28.663134543s) [noise=true , address=79.127.201.215:4358 , piece_id=2ZGKHBGSRVFZAXKJEN3GBZCEJVPEIHXZ2ZIAARFJRJRSQQBK5XDA, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:13.954258922 (35m29.7351923s) [noise=true , address=64.49.13.138:4483 , piece_id=5WKBRLGZZV43P3AMSSUVRWWIYSF54MYZZ33AVQVV463WLYBGP26A, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:16.171379307 (35m31.952312689s) [noise=true , address=45.91.116.11:4146 , piece_id=AQPQUYI3YCZC23ALHTLJYZ4YGWZ6XRQMS3PH62F63EV7YASALXHQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: storage node overloaded, request limit: 2000 Jul 26 22:27:16.935776619 (35m32.716709998s) [noise=true , address=45.91.116.5:4310 , piece_id=D4PIYEMA7PFEUESZ7BPNOHPEONWQKBRL2BGWIHG366PFV77MPGOA, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:27:18.454183596 (35m34.235116964s) [noise=true , address=45.91.116.14:4222 , piece_id=4YKN5ZIM2N2FV53GO3H7S6JHGIMSN5KDFV3VKHIHCXRMAKILH67A, seg_pos={0 0}, upload=555]: Done uploading piece. err: <nil> Jul 26 22:28:08.478930656 (36m24.259864043s) [noise=true , address=104.36.82.52:4090 , piece_id=DC6RDKRDWHE3GI7VCMOGHV5YH5ACOJ5UQRWNXJ46IVYEBSLPHG5Q, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: rpc: dial tcp 104.36.82.52:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479010621 (36m24.259943999s) [noise=true , address=104.36.82.4:4827 , piece_id=AOIJVARJ5LRFWLGTP24ZBT2MGQTER2JKQIH4XDO6MKPH4I2OUFAQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:1VXFTn1urnJ2qkUcDc5785hJ8L6bNVQTtgcgNeWLaZ9wEkKYVx, address:104.36.82.4:4827): protocol: rpc: dial tcp 104.36.82.4:4827: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479154384 (36m24.260087765s) [noise=true , address=104.36.82.21:4499 , piece_id=PRNLGOYXS3BQJEJAXAO6QJD4O4LQUFQYKZQEJDGKVBF742HJAESA, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:1218y8DtJNTTu94dbqGhywmHR4s99zcz9issmMmpgwusr2KgZxc, address:104.36.82.21:4499): protocol: rpc: dial tcp 104.36.82.21:4499: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479228428 (36m24.260161806s) [noise=true , address=104.36.82.56:4234 , piece_id=6E6KBP5GCJHYZIAFEP5J4RWR7TJ5SV5AFSA6FRWLNUSAYAVEYMTQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: rpc: dial tcp 104.36.82.56:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479293591 (36m24.260226969s) [noise=true , address=104.36.82.51:4081 , piece_id=Y3UHTSYTIWFWRSQFFXTJYD3SI4DFCIB3QUWSEQHQAL7N3Y2KKSCQ, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: rpc: dial tcp 104.36.82.51:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479352354 (36m24.260285732s) [noise=true , address=104.36.82.16:4300 , piece_id=SH6UOAO5HJIBRNMA34PQ2YAXYLBVLTMGQ6ULB6IHKP3F4NAYWS7A, seg_pos={0 0}, upload=555]: Done uploading piece. err: ecclient: upload failed (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg, address:104.36.82.16:4300): protocol: write tcp 159.223.143.39:37764->104.36.82.16:4300: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:08.479375580 (36m24.260308966s) [seg_pos={0 0}, upload=555]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:28:08.514602766 (36m24.295536137s) [upload=555]: Done uploading segments... Jul 26 22:28:08.514630796 (36m24.295564173s) [upload=555]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.039363616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........X..X........XXX.XXX.X.XX..X..X......X.............] duration: 658.414768ms FAIL (2025-07-26 22:28:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 658.447023ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................................X........X.X.X..] duration: 5.359646695s FAIL (2025-07-26 22:27:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.359664834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [................XX..XXX..X..XX..X..XX.X....X....X........X..] duration: 417.475794ms FAIL (2025-07-26 22:27:33 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.519671ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 520.561921ms FAIL (2025-07-26 22:27:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 520.653861ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX] duration: 1m0.028741691s FAIL (2025-07-26 22:27:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 24.25156866s) [noise=true , address=64.49.13.155:4916 , piece_id=CKNLNHNRKHRWSUX2DC6C6UBW5W5EXF4JEVDG4N72YRGWPX6II4TA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.471225286 (34m24.252158674s) [noise=true , address=104.36.82.41:4765 , piece_id=LGW6BM4RIHV4ED5ADJFTCHHM7YJBPCE4KWDWU2HK2MAYKTIGN6JQ, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.471739427 (34m24.252672815s) [noise=true , address=104.36.82.52:4087 , piece_id=7ITOZLFRY63TNYADJFUMFJRONSCPLRTYC5W67SJ4YCBTB6N36W5A, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.472277321 (34m24.253210706s) [noise=true , address=104.36.82.16:4317 , piece_id=AFZUMZTZHG23ESTJKPHJVVZWPRCLZZ4LQPB6SHLND6MP4U3T63VA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.474588396 (34m24.255521784s) [noise=true , address=104.36.82.35:4589 , piece_id=PG4SK5QUQ6C6DIM3ROPRU5SOHFHEKSDVSHMMRYVWQ7POOQOIJQPQ, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.475071564 (34m24.256004954s) [noise=true , address=45.91.116.14:4222 , piece_id=7BZVGGSCHKIADBLFGRKTMDDLQM7SRKHASMWGXJGA3BMXRKPJIYHQ, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.475509276 (34m24.256442666s) [noise=true , address=79.127.205.245:4239 , piece_id=PWZQZMIEERREFGK2X7PQLM3HAT6NEVR5WK44WDMC7FTKEY6EY24Q, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.475965302 (34m24.256898687s) [noise=true , address=104.36.82.37:4677 , piece_id=A6VF5UCK4DJCKPGPLD5PNR4FS5XFR4NUYB2GJIYA5VXSGBYS23YA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.476394670 (34m24.257328054s) [noise=true , address=104.36.82.55:4228 , piece_id=KTKVQNLA3HRFHOC7VCAY36UKXJW57MDYSLJCWWDDRNETZ4K5QD4A, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.476801074 (34m24.25773446s) [noise=true , address=79.127.205.235:4689 , piece_id=N2KTACUFHS6YKN6IO564SVYKX52LXQC2NDP3IBCAM3KWIQE6UHSQ, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.477196730 (34m24.258130122s) [noise=true , address=79.127.205.234:4614 , piece_id=UUJ7SNHI3ARET6F5RSH4I4VQIDCX2B5J2EWTNRADVUUPM4MP2BWA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.477610913 (34m24.258544302s) [noise=true , address=45.91.116.9:4267 , piece_id=HLYAB64GJZPQCUCWM37Y57DCWCNAQB4E4HUQ3SIFLNBN2BVQT6QA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.477993168 (34m24.258926556s) [noise=true , address=79.127.220.101:4048 , piece_id=AJWNXMV7SFAKXM4VS75F725SWYERPFRUUDGGF6LNA56M6XHSVXJQ, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.478453454 (34m24.259386843s) [noise=true , address=104.36.82.56:4235 , piece_id=WFY35ZFBT7ZXWR2SDTTDPFCW7QAICXI3AGD5PXNAK5JUG7J3TZCA, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.480710981 (34m24.261644369s) [noise=true , address=104.36.82.5:4872 , piece_id=NHTYHU3X2UMBJJEPK7VMQOKLBLTFLRDSOU52KYXQ62IITAKDBS5A, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.481248617 (34m24.262182007s) [noise=true , address=77.90.31.17:4796 , piece_id=44LDCR6YJGFZPMBCVYOFTKCEH5ZN2JGHRTCQ2UHMHR6TF5D4HE2Q, seg_pos={0 0}, upload=538]: Uploading piece... Jul 26 22:26:08.638997039 (34m24.419930419s) [noise=true , address=104.36.82.55:4228 , piece_id=KTKVQNLA3HRFHOC7VCAY36UKXJW57MDYSLJCWWDDRNETZ4K5QD4A, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: storage node overloaded, request limit: 2000 Jul 26 22:26:08.649206875 (34m24.430140266s) [upload=538]: Next returned nil segment Jul 26 22:26:08.649229271 (34m24.430162635s) [upload=538]: Waiting for error group managing segments... Jul 26 22:26:08.962307497 (34m24.743240867s) [noise=true , address=79.127.220.101:4048 , piece_id=AJWNXMV7SFAKXM4VS75F725SWYERPFRUUDGGF6LNA56M6XHSVXJQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:09.110679643 (34m24.89161301s) [noise=true , address=79.127.205.234:4614 , piece_id=UUJ7SNHI3ARET6F5RSH4I4VQIDCX2B5J2EWTNRADVUUPM4MP2BWA, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:09.206468649 (34m24.987402024s) [noise=true , address=64.49.13.155:4916 , piece_id=CKNLNHNRKHRWSUX2DC6C6UBW5W5EXF4JEVDG4N72YRGWPX6II4TA, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:09.385442736 (34m25.166376114s) [noise=true , address=79.127.205.235:4689 , piece_id=N2KTACUFHS6YKN6IO564SVYKX52LXQC2NDP3IBCAM3KWIQE6UHSQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:09.673810066 (34m25.454743447s) [noise=true , address=79.127.205.245:4239 , piece_id=PWZQZMIEERREFGK2X7PQLM3HAT6NEVR5WK44WDMC7FTKEY6EY24Q, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:11.217259182 (34m26.998192544s) [noise=true , address=45.91.116.9:4267 , piece_id=HLYAB64GJZPQCUCWM37Y57DCWCNAQB4E4HUQ3SIFLNBN2BVQT6QA, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:26:17.945117750 (34m33.726051129s) [noise=true , address=45.91.116.14:4222 , piece_id=7BZVGGSCHKIADBLFGRKTMDDLQM7SRKHASMWGXJGA3BMXRKPJIYHQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: <nil> Jul 26 22:27:08.393443389 (35m24.174376773s) [noise=true , address=104.36.82.16:4317 , piece_id=AFZUMZTZHG23ESTJKPHJVVZWPRCLZZ4LQPB6SHLND6MP4U3T63VA, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1FdJoeW4vejiQx1QkQ1nCkApgQaW6fB66d2usAYVfFuK1PPmRP, address:104.36.82.16:4317): protocol: rpc: dial tcp 104.36.82.16:4317: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393544825 (35m24.174478209s) [noise=true , address=104.36.82.5:4872 , piece_id=NHTYHU3X2UMBJJEPK7VMQOKLBLTFLRDSOU52KYXQ62IITAKDBS5A, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1zqxiFpqKeXT9mhXUDuZZmvJkhejC8gcHTQgTadY1xUpQyLyZi, address:104.36.82.5:4872): protocol: rpc: dial tcp 104.36.82.5:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393625675 (35m24.174559051s) [noise=true , address=104.36.82.52:4087 , piece_id=7ITOZLFRY63TNYADJFUMFJRONSCPLRTYC5W67SJ4YCBTB6N36W5A, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: rpc: dial tcp 104.36.82.52:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393695046 (35m24.174628423s) [noise=true , address=104.36.82.37:4677 , piece_id=A6VF5UCK4DJCKPGPLD5PNR4FS5XFR4NUYB2GJIYA5VXSGBYS23YA, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:12QgDye4pcKhsndRP7tCxM44KZkSyRnhp219u8JnRHXX7jRutPv, address:104.36.82.37:4677): protocol: rpc: dial tcp 104.36.82.37:4677: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393763323 (35m24.1746967s) [noise=true , address=104.36.82.42:4819 , piece_id=CA3A7OBKHQGMPP44JSEKHXI6KXI7WLN6JONUEFK54KOOMY72CDPQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:12MZhrDu1fjMN7XFoR2h1q3JPLHJYYURGQwQgcgLuucSrGShjoS, address:104.36.82.42:4819): protocol: rpc: dial tcp 104.36.82.42:4819: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393831356 (35m24.174764734s) [noise=true , address=104.36.82.12:4181 , piece_id=G7XRU7P3CYATUTJTEDYDLW7LAQFRF47X7ABNK25H25SD4XDVK4YQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1XH92Sdi4ZiRg1iic5A9Mp8RyKZsk9WfVQHLoQ8t9Vvk2Lyxwh, address:104.36.82.12:4181): protocol: rpc: dial tcp 104.36.82.12:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393896920 (35m24.174830297s) [noise=true , address=77.90.31.17:4796 , piece_id=44LDCR6YJGFZPMBCVYOFTKCEH5ZN2JGHRTCQ2UHMHR6TF5D4HE2Q, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: rpc: dial tcp 77.90.31.17:4796: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.393960559 (35m24.174893938s) [noise=true , address=104.36.82.41:4765 , piece_id=LGW6BM4RIHV4ED5ADJFTCHHM7YJBPCE4KWDWU2HK2MAYKTIGN6JQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:104.36.82.41:4765): protocol: rpc: dial tcp 104.36.82.41:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.394024616 (35m24.174957993s) [noise=true , address=104.36.82.35:4589 , piece_id=PG4SK5QUQ6C6DIM3ROPRU5SOHFHEKSDVSHMMRYVWQ7POOQOIJQPQ, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1uKzu7nZ3GTxPz8m8JFAtDR6Yvj2UQwNdvQGt4JnY672KGEkES, address:104.36.82.35:4589): protocol: rpc: dial tcp 104.36.82.35:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.394096792 (35m24.175030176s) [noise=true , address=104.36.82.56:4235 , piece_id=WFY35ZFBT7ZXWR2SDTTDPFCW7QAICXI3AGD5PXNAK5JUG7J3TZCA, seg_pos={0 0}, upload=538]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: write tcp 159.223.143.39:47950->104.36.82.56:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:08.394122351 (35m24.175055739s) [seg_pos={0 0}, upload=538]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:27:08.418086562 (35m24.199019935s) [upload=538]: Done uploading segments... Jul 26 22:27:08.418115856 (35m24.199049233s) [upload=538]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.028770564s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 559.755213ms FAIL (2025-07-26 22:27:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 559.793724ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [...........X..X........XXX.XXX.X.XX..X..X......X............] duration: 502.187842ms FAIL (2025-07-26 22:26:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 502.212745ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X........X.X.X.] duration: 11.34738458s FAIL (2025-07-26 22:26:43 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.347413348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X........X.X.X] duration: 1m0.001009426s FAIL (2025-07-26 22:26:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001044754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.................XX..XXX..X..XX..X..XX.X....X....X........X.] duration: 604.067406ms FAIL (2025-07-26 22:26:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 604.101915ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration: 1m0.033500006s FAIL (2025-07-26 22:26:03 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== WY5JV5A, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.242126122 (33m19.023059512s) [noise=true , address=79.127.219.47:4069 , piece_id=IFDIWTMBWAJDEFZFUJCX53VVU2MTJZTE5J5STYCKT6OAFT6UUGKA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.242356864 (33m19.023290252s) [noise=true , address=64.49.13.140:4508 , piece_id=7H3YCEUT26RWXLJUV5EQSQRG6SBUX4K5PI3THT7FPYUQXGEZS3CA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.242562140 (33m19.023495521s) [noise=true , address=104.36.82.55:4228 , piece_id=FGXP2NNTU5IATOQKF6FHPXTO4A4NA3N7BCIFWM3IQESGGSF64UBQ, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.242912896 (33m19.023846281s) [noise=true , address=79.127.219.38:4004 , piece_id=MSLWHKRKD2ZXPXU73HAEXYU7SESJ4SIEYOLN2CHPUCQYOP3WA5NA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.242946351 (33m19.023879741s) [noise=true , address=104.36.82.56:4234 , piece_id=OPYBCBF4AJPAUHMP5UCO4WBM7RJ2RH7ZEHOGAWTDSPGKFKXVLAZA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.243332587 (33m19.024265969s) [noise=true , address=77.90.31.15:4726 , piece_id=3EJGMZIWPWBAVSA6AZPDRO4XNCXDN3QFZNGI5NAJA5DSYHXKRUXQ, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.243556391 (33m19.024489777s) [noise=true , address=104.36.82.13:4199 , piece_id=2WM67WHZZLFCUU42RYWARGL64IMLNXNL7RNXE7CA75HPCFVLOD6A, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.247501553 (33m19.028434941s) [noise=true , address=77.90.31.17:4796 , piece_id=PIDEYKX37B4DDG3S2W5JCUOGXGUSZF57RCADSY3FESNJVIL4MVPA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.248246455 (33m19.029179832s) [noise=true , address=77.90.31.11:4560 , piece_id=DCQGCLCXOEVCNF3YTHFRATEAFL47GEJTWRANKIKRDH6TGIYIXIIQ, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.248735549 (33m19.029668935s) [noise=true , address=104.36.82.7:4922 , piece_id=5MLUACJA4KYMIHUHL5M4RF2WSEGORQCPDQTC4X46NZ2QVYQBLK7A, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.248852573 (33m19.029785961s) [noise=true , address=104.36.82.2:4755 , piece_id=V4EX5DHU2FBKS6PBGHDKVVVWLX2IPHFE3PDVTBLEWU2ANQQ5K27A, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.249397978 (33m19.030331365s) [noise=true , address=45.91.116.9:4267 , piece_id=P7A3F5LKHBUZ6IP3WTVLOSWYZ73QK7XSLFMF44YSWPVSTEM2EOGA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.249862061 (33m19.03079545s) [noise=true , address=104.36.82.51:4084 , piece_id=UEE5PVLTDFFAIPEBUZXP5VUILHX5FSNICCREOTDU4MWJIX74JUJA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.250461925 (33m19.031395321s) [noise=true , address=104.36.82.11:4146 , piece_id=TXPTK6Q4Y5ZE4IVEDMNXM4YTTB6CIA3VN7ZUFQQALIE5MIIEP6KA, seg_pos={0 0}, upload=522]: Uploading piece... Jul 26 22:25:03.335517996 (33m19.116451388s) [noise=true , address=79.127.219.47:4069 , piece_id=IFDIWTMBWAJDEFZFUJCX53VVU2MTJZTE5J5STYCKT6OAFT6UUGKA, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:18dK6RxbMxU4a46pW4YPiQW87xgpeoTb1ouqE9fzTdzTAFAa3W, address:79.127.219.47:4069): protocol: storage node overloaded, request limit: 2000 Jul 26 22:25:03.362778206 (33m19.143711584s) [upload=522]: Next returned nil segment Jul 26 22:25:03.362802210 (33m19.143735597s) [upload=522]: Waiting for error group managing segments... Jul 26 22:25:03.365336869 (33m19.146270233s) [noise=true , address=79.127.219.38:4004 , piece_id=MSLWHKRKD2ZXPXU73HAEXYU7SESJ4SIEYOLN2CHPUCQYOP3WA5NA, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:25:03.372034817 (33m19.152968181s) [noise=true , address=45.91.116.9:4267 , piece_id=P7A3F5LKHBUZ6IP3WTVLOSWYZ73QK7XSLFMF44YSWPVSTEM2EOGA, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:25:03.401661243 (33m19.182594607s) [noise=true , address=77.90.31.11:4560 , piece_id=DCQGCLCXOEVCNF3YTHFRATEAFL47GEJTWRANKIKRDH6TGIYIXIIQ, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:25:03.526068921 (33m19.307002291s) [noise=true , address=77.90.31.15:4726 , piece_id=3EJGMZIWPWBAVSA6AZPDRO4XNCXDN3QFZNGI5NAJA5DSYHXKRUXQ, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:25:03.727396258 (33m19.50832964s) [noise=true , address=104.36.82.55:4228 , piece_id=FGXP2NNTU5IATOQKF6FHPXTO4A4NA3N7BCIFWM3IQESGGSF64UBQ, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: storage node overloaded, request limit: 2000 Jul 26 22:25:04.363269378 (33m20.144202762s) [noise=true , address=64.49.13.140:4508 , piece_id=7H3YCEUT26RWXLJUV5EQSQRG6SBUX4K5PI3THT7FPYUQXGEZS3CA, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:25:05.662641144 (33m21.443574531s) [noise=true , address=64.49.13.169:4716 , piece_id=CUIDRZDYM3XWO4UOTF3YDHEKPMBTAOTNG3OV63BKUXSMOBEUBQNQ, seg_pos={0 0}, upload=522]: Done uploading piece. err: <nil> Jul 26 22:26:03.141300303 (34m18.922233688s) [noise=true , address=104.36.82.40:4953 , piece_id=755HXZWUZXT4RKFQF3NSSJPTLKKQWO4O2NBRU6LPTKJU5MNAET6A, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:12dhNW98YkWKWzXBnwmTCvzaw9CtvxRW8ELJnAjHnATFhpM1oKn, address:104.36.82.40:4953): protocol: rpc: dial tcp 104.36.82.40:4953: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141414422 (34m18.9223478s) [noise=true , address=104.36.82.56:4234 , piece_id=OPYBCBF4AJPAUHMP5UCO4WBM7RJ2RH7ZEHOGAWTDSPGKFKXVLAZA, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: write tcp 159.223.143.39:48556->104.36.82.56:4234: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141479294 (34m18.922412673s) [noise=true , address=104.36.82.2:4755 , piece_id=V4EX5DHU2FBKS6PBGHDKVVVWLX2IPHFE3PDVTBLEWU2ANQQ5K27A, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: rpc: dial tcp 104.36.82.2:4755: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141548226 (34m18.922481603s) [noise=true , address=77.90.31.17:4796 , piece_id=PIDEYKX37B4DDG3S2W5JCUOGXGUSZF57RCADSY3FESNJVIL4MVPA, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 159.223.143.39:32880->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141618263 (34m18.922551641s) [noise=true , address=104.36.82.5:4854 , piece_id=BTT3UUHTR2JCQ7473Y4ZWGS6JAF7VQFADW2HKOBBR4EOJWY5JV5A, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy, address:104.36.82.5:4854): protocol: rpc: dial tcp 104.36.82.5:4854: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141682694 (34m18.922616074s) [noise=true , address=104.36.82.11:4146 , piece_id=TXPTK6Q4Y5ZE4IVEDMNXM4YTTB6CIA3VN7ZUFQQALIE5MIIEP6KA, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:1Sv7eCy62QTiVT1c7mxzJSTAnrCsDoRrxqxfL6govEFRJRQJ6V, address:104.36.82.11:4146): protocol: rpc: dial tcp 104.36.82.11:4146: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141742715 (34m18.922676102s) [noise=true , address=104.36.82.42:4817 , piece_id=EVFDOVL3OIPCCXAGBCMHVLNZ2IRIRV7JLGZESNXMJ3LUVKRPDSTQ, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:1w1W2uV2kFMfxwSwB5o1rZbd4VGVR2XA7sk3NLWRQcZoBnheE7, address:104.36.82.42:4817): protocol: rpc: dial tcp 104.36.82.42:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141807146 (34m18.922740633s) [noise=true , address=104.36.82.13:4199 , piece_id=2WM67WHZZLFCUU42RYWARGL64IMLNXNL7RNXE7CA75HPCFVLOD6A, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:1DagQMPpe9F6w5xewXXCfzVBPcmgg6EKCauFsNCWHoHdP56bs7, address:104.36.82.13:4199): protocol: write tcp 159.223.143.39:32852->104.36.82.13:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141873320 (34m18.922806698s) [noise=true , address=104.36.82.7:4922 , piece_id=5MLUACJA4KYMIHUHL5M4RF2WSEGORQCPDQTC4X46NZ2QVYQBLK7A, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:12KrMgT4Pwj9UFE88q6tt65D7o6q68NhsmEXmifNmvzJ6eEkm8R, address:104.36.82.7:4922): protocol: write tcp 159.223.143.39:40742->104.36.82.7:4922: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141935463 (34m18.922868845s) [noise=true , address=104.36.82.51:4084 , piece_id=UEE5PVLTDFFAIPEBUZXP5VUILHX5FSNICCREOTDU4MWJIX74JUJA, seg_pos={0 0}, upload=522]: Done uploading piece. err: ecclient: upload failed (node:121BaY9pfRq2a2sZTqmPA3NrXNh9pnckJj21N13sV4jhkuo2gP9, address:104.36.82.51:4084): protocol: write tcp 159.223.143.39:44906->104.36.82.51:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:03.141956884 (34m18.922890269s) [seg_pos={0 0}, upload=522]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 22:26:03.169669305 (34m18.950602669s) [upload=522]: Done uploading segments... Jul 26 22:26:03.169699803 (34m18.95063317s) [upload=522]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.033538616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [............X..X........XXX.XXX.X.XX..X..X......X...........] duration: 570.779913ms FAIL (2025-07-26 22:26:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 570.806768ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 593.92687ms FAIL (2025-07-26 22:25:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 593.956464ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..................XX..XXX..X..XX..X..XX.X....X....X........X] duration: 30.030608824s FAIL (2025-07-26 22:25:52 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.030649121s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 582.322755ms FAIL (2025-07-26 22:25:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 582.388072ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.............X..X........XXX.XXX.X.XX..X..X......X..........] duration: 381.730998ms FAIL (2025-07-26 22:25:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 381.763381ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X........X.X.] duration: 11.921396468s FAIL (2025-07-26 22:25:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.921412611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 516.385127ms FAIL (2025-07-26 22:25:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 516.419914ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................................X........X.X] duration: 1m0.001460018s FAIL (2025-07-26 22:25:08 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001487852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 1m0.030954529s FAIL (2025-07-26 22:25:03 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .083214491 (32m18.864147881s) [upload=503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:24:03.139157332 (32m18.92009072s) [seg_pos={0 0}, upload=503]: Begin upload segment... Jul 26 22:24:03.139212475 (32m18.920145854s) [seg_pos={0 0}, upload=503]: Done begin upload segment. Jul 26 22:24:03.139248160 (32m18.920181546s) [noise=true , address=64.49.13.154:4935 , piece_id=CLT6A2GU65HOCCQK7XBSN7JPIQADSUGMSFZCWSMO4AIJDADVVT5A, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.140079895 (32m18.921013284s) [noise=true , address=104.36.82.56:4229 , piece_id=NLEAS7Q3IQJ5FXXSJ4TRBDVCFVEWZHCMFCK6UX2SJBL3NGJSK3DQ, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.140692424 (32m18.921625813s) [noise=true , address=104.36.82.11:4126 , piece_id=ZZFILS23JQIHI753WDC7CIV3JP6RUCZ7I5CLUSGQYOY3KYRGVMLA, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.141369994 (32m18.922303384s) [noise=true , address=104.36.82.41:4792 , piece_id=BDJU5UKDDY6QOFN7L6NKQUT35VV24E4YZVBC6V2HUIWDC2L6ANNQ, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.142116568 (32m18.92304996s) [noise=true , address=45.91.116.14:4206 , piece_id=36W7JCG2XUXLQPIHEQ5XRAAHDZWOQYTIFEBT7HE4SY3IISQ7JFJA, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.142579117 (32m18.923512506s) [noise=true , address=79.127.219.47:4061 , piece_id=Y6W6436Z36DRUHNR5ASL5IZODXPKIEPWMAZ6Q6ESUTZF2PYQHCAQ, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.143125880 (32m18.92405927s) [noise=true , address=79.127.220.101:4048 , piece_id=K4TP4FQLZ2E23ZBB5E5ZAEWCNLZZBQGO6PP67HXYTCE46WO5DPIA, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.143557890 (32m18.924491279s) [noise=true , address=79.127.205.235:4691 , piece_id=PT4LA2LDZD7YBMPANXVG6MOFU6NQNGUMTI5AWYPLPQ2IWQ6YMOPA, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.144034786 (32m18.924968176s) [noise=true , address=79.127.205.245:4246 , piece_id=PHWJA47NWJLCTLEBFPO3F36PVEHGBH6656WVDXEQKNSBIT5Q2IVQ, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.144595070 (32m18.925528462s) [noise=true , address=104.36.82.44:4900 , piece_id=PETKKOB2SSEN33LR5LN7TZWL4XRXRYNGDZ5LZM2ZNZ75MA4MRQOQ, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.145117543 (32m18.926050933s) [noise=true , address=104.36.82.51:4080 , piece_id=3UJWSWYGROZ5HSPKKLHRLNW5XY2F6UD6GXT5PRVLPLZAH2Z52LLA, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.145613430 (32m18.926546814s) [noise=true , address=79.127.205.234:4591 , piece_id=URYYMBA3WBBQIVPJU2JII2T5N2BMQDDXWQV3NWDWTXYQEOIQFJTA, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.145963666 (32m18.926897048s) [noise=true , address=79.127.219.38:4009 , piece_id=LIAQLZ7Q3LEFGGPXNNB7MYYI2HTHEAKM6OQGC4NA6H6VSJHTHIOQ, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.146288984 (32m18.927222373s) [noise=true , address=79.127.201.215:4353 , piece_id=4UD5MICGS5RGHABBWBGPXTRLNQOFRJ647FEHTGQFXJAMQBK6ARRQ, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.146805432 (32m18.927738824s) [noise=true , address=79.127.205.246:4288 , piece_id=UOKXQ5UIMSRLOHIPLHARU4L2UBPVIYB2XNO5VQKTFSZBIA325UOQ, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.152009855 (32m18.932943248s) [noise=true , address=104.36.82.12:4183 , piece_id=TDVU5BCO65XQ65KXAPPKXWF7WCZTBJCB2EXP7R2SU7Y35HJIHOMA, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.153257403 (32m18.934190793s) [noise=true , address=45.91.116.5:4310 , piece_id=I2YDYKVTRY273IK7T6XKT5CF77XXWKGUSUPLT4T4MDB2MLNS6FKA, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.154200639 (32m18.935134036s) [noise=true , address=77.90.31.21:4910 , piece_id=RDWPQ2XIAPUBELKUT3GCTNIDHLM2OXBOL7N3AOOWE4X5QK73OA6Q, seg_pos={0 0}, upload=503]: Uploading piece... Jul 26 22:24:03.264315079 (32m19.045248465s) [noise=true , address=79.127.219.47:4061 , piece_id=Y6W6436Z36DRUHNR5ASL5IZODXPKIEPWMAZ6Q6ESUTZF2PYQHCAQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:24:03.351122179 (32m19.132055543s) [upload=503]: Next returned nil segment Jul 26 22:24:03.351146316 (32m19.1320797s) [upload=503]: Waiting for error group managing segments... Jul 26 22:24:03.374683748 (32m19.155617113s) [noise=true , address=79.127.219.38:4009 , piece_id=LIAQLZ7Q3LEFGGPXNNB7MYYI2HTHEAKM6OQGC4NA6H6VSJHTHIOQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Jul 26 22:24:03.770580897 (32m19.551514283s) [noise=true , address=79.127.205.234:4591 , piece_id=URYYMBA3WBBQIVPJU2JII2T5N2BMQDDXWQV3NWDWTXYQEOIQFJTA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Jul 26 22:24:03.790895807 (32m19.571829188s) [noise=true , address=79.127.205.246:4288 , piece_id=UOKXQ5UIMSRLOHIPLHARU4L2UBPVIYB2XNO5VQKTFSZBIA325UOQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Jul 26 22:24:03.802220120 (32m19.58315349s) [noise=true , address=79.127.205.235:4691 , piece_id=PT4LA2LDZD7YBMPANXVG6MOFU6NQNGUMTI5AWYPLPQ2IWQ6YMOPA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Jul 26 22:24:03.804975700 (32m19.585909064s) [noise=true , address=79.127.205.245:4246 , piece_id=PHWJA47NWJLCTLEBFPO3F36PVEHGBH6656WVDXEQKNSBIT5Q2IVQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Jul 26 22:24:03.850303337 (32m19.631236727s) [noise=true , address=64.49.13.154:4935 , piece_id=CLT6A2GU65HOCCQK7XBSN7JPIQADSUGMSFZCWSMO4AIJDADVVT5A, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12GZpRx4CMfeRa5nUrXmBaTUxcRVfLkx6v4StbcVyWpVqeuZdfk, address:64.49.13.154:4935): protocol: storage node overloaded, request limit: 2000 Jul 26 22:24:05.850482745 (32m21.631416122s) [noise=true , address=45.91.116.5:4310 , piece_id=I2YDYKVTRY273IK7T6XKT5CF77XXWKGUSUPLT4T4MDB2MLNS6FKA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Jul 26 22:24:08.249120240 (32m24.030053609s) [noise=true , address=79.127.201.215:4353 , piece_id=4UD5MICGS5RGHABBWBGPXTRLNQOFRJ647FEHTGQFXJAMQBK6ARRQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Jul 26 22:24:10.183910415 (32m25.964843778s) [noise=true , address=45.91.116.14:4206 , piece_id=36W7JCG2XUXLQPIHEQ5XRAAHDZWOQYTIFEBT7HE4SY3IISQ7JFJA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Jul 26 22:24:10.721375215 (32m26.502308596s) [noise=true , address=79.127.220.101:4048 , piece_id=K4TP4FQLZ2E23ZBB5E5ZAEWCNLZZBQGO6PP67HXYTCE46WO5DPIA, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Jul 26 22:24:29.037369330 (32m44.818302693s) [noise=true , address=77.90.31.21:4910 , piece_id=RDWPQ2XIAPUBELKUT3GCTNIDHLM2OXBOL7N3AOOWE4X5QK73OA6Q, seg_pos={0 0}, upload=503]: Done uploading piece. err: <nil> Jul 26 22:25:03.022389188 (33m18.803322572s) [noise=true , address=104.36.82.56:4229 , piece_id=NLEAS7Q3IQJ5FXXSJ4TRBDVCFVEWZHCMFCK6UX2SJBL3NGJSK3DQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:104.36.82.56:4229): protocol: rpc: dial tcp 104.36.82.56:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:03.022496991 (33m18.803430373s) [noise=true , address=104.36.82.11:4126 , piece_id=ZZFILS23JQIHI753WDC7CIV3JP6RUCZ7I5CLUSGQYOY3KYRGVMLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12R3SVXrhJsmYTXR4vDSkfaiqDRox21u17SLmQc9sEab1B7zZRr, address:104.36.82.11:4126): protocol: rpc: dial tcp 104.36.82.11:4126: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:03.022573983 (33m18.803507352s) [noise=true , address=104.36.82.41:4792 , piece_id=BDJU5UKDDY6QOFN7L6NKQUT35VV24E4YZVBC6V2HUIWDC2L6ANNQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: rpc: dial tcp 104.36.82.41:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:03.022647886 (33m18.803581255s) [noise=true , address=104.36.82.51:4080 , piece_id=3UJWSWYGROZ5HSPKKLHRLNW5XY2F6UD6GXT5PRVLPLZAH2Z52LLA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C, address:104.36.82.51:4080): protocol: rpc: dial tcp 104.36.82.51:4080: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:03.022729841 (33m18.80366322s) [noise=true , address=104.36.82.12:4183 , piece_id=TDVU5BCO65XQ65KXAPPKXWF7WCZTBJCB2EXP7R2SU7Y35HJIHOMA, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:104.36.82.12:4183): protocol: rpc: dial tcp 104.36.82.12:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:03.022804722 (33m18.803738105s) [noise=true , address=104.36.82.44:4900 , piece_id=PETKKOB2SSEN33LR5LN7TZWL4XRXRYNGDZ5LZM2ZNZ75MA4MRQOQ, seg_pos={0 0}, upload=503]: Done uploading piece. err: ecclient: upload failed (node:1R8K65zsCspu7W2oAnvtpz7zEsQFkDG1e5BAp6faQYMQrkh4oQ, address:104.36.82.44:4900): protocol: write tcp 159.223.143.39:59270->104.36.82.44:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:03.022828900 (33m18.803762288s) [seg_pos={0 0}, upload=503]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:25:03.049318586 (33m18.830251964s) [upload=503]: Done uploading segments... Jul 26 22:25:03.049352757 (33m18.830286147s) [upload=503]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration:* 1m0.03097668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...................XX..XXX..X..XX..X..XX.X....X....X........] duration: 419.149164ms FAIL (2025-07-26 22:24:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716601656 (24m59.497535035s) [noise=true , address=77.90.31.17:4770 , piece_id=PSP7DMQDRJN36NVG3HBGHS6SQ2L24XUP5UKWYFS4DDIMNIAGLUQA, seg_pos={0 0}, upload=387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Jul 26 22:16:43.716626163 (24m59.497559539s) [seg_pos={0 0}, upload=387]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 26 22:16:43.716716659 (24m59.497650035s) [upload=387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:16:43.716721768 (24m59.497655146s) [upload=387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:16:43.860689722 (24m59.641623091s) [upload=387]: Done uploading segments... Jul 26 22:16:43.860712707 (24m59.641646088s) [upload=387]: Upload finished. err: <nil> Jul 26 22:17:18.858881116 (25m34.6398146s) [upload=398]: Starting upload Jul 26 22:17:18.858970970 (25m34.639904352s) [upload=398]: Done starting upload Jul 26 22:17:18.859096325 (25m34.640029691s) [upload=398]: Uploading segments... Jul 26 22:17:18.859347960 (25m34.640281336s) [upload=398]: Got next segment. Inline: false Jul 26 22:17:18.859409331 (25m34.640342708s) [upload=398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:17:18.859416079 (25m34.640349476s) [upload=398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:17:18.891826819 (25m34.672760195s) [seg_pos={0 0}, upload=398]: Begin upload segment... Jul 26 22:17:18.891907216 (25m34.672840598s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.892618737 (25m34.673552121s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893062440 (25m34.673995819s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893460552 (25m34.674393935s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.893838747 (25m34.674772128s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894241274 (25m34.675174656s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.894452461 (25m34.675385837s) [seg_pos={0 0}, upload=398]: Done begin upload segment. Jul 26 22:17:18.894483833 (25m34.675417214s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895251136 (25m34.676184517s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.895667584 (25m34.676600966s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.896057589 (25m34.676990964s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.897411403 (25m34.67834479s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.898569054 (25m34.679502433s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.899319352 (25m34.680252734s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900381012 (25m34.68131439s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.900815872 (25m34.681749251s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:18.901686119 (25m34.682619504s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Uploading piece... Jul 26 22:17:19.000254924 (25m34.781188329s) [upload=398]: Next returned nil segment Jul 26 22:17:19.000472983 (25m34.781406409s) [upload=398]: Waiting for error group managing segments... Jul 26 22:17:19.005490516 (25m34.786423887s) [noise=true , address=45.91.116.10:4081 , piece_id=EJCATPYFLWS3TZWJXWKN2WIMF3Q6WYFOIKQZCEGVP37TVKVSCGEA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.005571749 (25m34.786505117s) [noise=true , address=45.91.116.8:4045 , piece_id=XEEJ3WFGRZDUMF3G2URXSSRPWV62LCSE53HWLQGY3TVZTH7A2Q3Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006165546 (25m34.787098912s) [noise=true , address=45.91.116.4:4939 , piece_id=NJCQX4RG3HKHW3DZRLWUHO5HB6IOT65TEJJT5YG22CEMPOFUIPXQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006482334 (25m34.78741574s) [noise=true , address=45.91.116.13:4155 , piece_id=5HES6JEZ4QY3DM4XEW34AKLR6TCCGXVAVUVQMTXSGGOBCVHBAYHQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.006710744 (25m34.78764411s) [noise=true , address=45.91.116.6:4975 , piece_id=2CNQF5THI4YLUS2JIUCGXJCK3CYVSQOWD67ZUZSPPO2BNI2OBB4A, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007080151 (25m34.788013515s) [noise=true , address=45.91.116.9:4261 , piece_id=35YFP2W53AKX5X3F4LIXJHR4IUQ723B6IIRFXYLVNVWKXP46FZZA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007202058 (25m34.788135424s) [noise=true , address=45.91.116.5:4298 , piece_id=WXY72TNVZ5QEFTYFZYDISI7IZZZUUPXEMTMVFKGMFJGQHYNHDVFQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007705224 (25m34.788638596s) [noise=true , address=45.91.116.10:4083 , piece_id=7LSIVIYCHZNVEQ3JGJHYFWDTRASQJ5QA3CIMOCZCRIACWKYTF6BA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.007716193 (25m34.788649589s) [noise=true , address=45.91.116.6:4974 , piece_id=COPWQHHS4VBMC6DHXIRIX7DZEEILX3XD32UMX22RJI25E7XDKWFA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.008533498 (25m34.789466881s) [noise=true , address=45.91.116.7:4009 , piece_id=XFUVFBAZSKMKENGJ4F66H2URRRECNVFK2BKHI7STRDTEKLSVHHLQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.010437757 (25m34.791371145s) [noise=true , address=77.90.31.14:4661 , piece_id=XP7P7KNCL65G5MNQOWLZ273WVHZP6DX7SRPAA7MHUW66E2QAZ4ZQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:19.025360018 (25m34.80629342s) [noise=true , address=77.90.31.21:4877 , piece_id=BYBOOLA2NCT5UWYK2ZF3YYN46ZPNAX6B4BVSEHOVILO4ZQTYJMTA, seg_pos={0 0}, upload=398]: Done uploading piece. err: <nil> Jul 26 22:17:48.826740557 (26m4.607673933s) [noise=true , address=104.36.82.3:4767 , piece_id=H7FM2BSECPB5VIVMYTE6QMYNDOYSOJ63N4PJRSFFDNEK6K3SUYPA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:44626->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826946196 (26m4.607879571s) [noise=true , address=104.36.82.13:4186 , piece_id=PVEPZ2QUZ7RQDR3WP2YNGO3L3WBADPJSG72TONTJ5N4GKTZMSPMA, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.826956281 (26m4.607889648s) [noise=true , address=104.36.82.10:4079 , piece_id=HTDBXKD6XB4GJZKLIOEGHCXI25ORSWKPOWFX3J2FKT5RUA4376RQ, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827002852 (26m4.607936227s) [noise=true , address=104.36.82.16:4295 , piece_id=6KQQYY4ETRXXQ34TVFD67T3UJCZDVGL7GIJ4FYUQSR4XWLG7DE7Q, seg_pos={0 0}, upload=398]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:36584->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:17:48.827063928 (26m4.607997303s) [seg_pos={0 0}, upload=398]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:17:48.857413352 (26m4.638346721s) [upload=398]: Done uploading segments... Jul 26 22:17:48.857443022 (26m4.638376406s) [upload=398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 419.179605ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 625.384994ms FAIL (2025-07-26 22:24:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 625.425274ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............X..X........XXX.XXX.X.XX..X..X......X.........] duration: 506.515333ms FAIL (2025-07-26 22:24:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 506.549044ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [...............X..X........XXX.XXX.X.XX..X..X......X........] duration: 440.417233ms FAIL (2025-07-26 22:24:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 440.449885ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)