Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [........XXX.XXX.X.XX..X..X......X.............X.............] duration: 636.568534ms FAIL (2025-07-26 22:38:19 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 636.69584ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....XX..XXX..X..XX..X..XX.X....X....X........X..............] duration: 672.576637ms FAIL (2025-07-26 22:38:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 672.617658ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.] duration: 15.484074278s FAIL (2025-07-26 22:38:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 22:36:43.901527855 (44m59.682461222s) [upload=691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:36:43.901542041 (44m59.682475408s) [upload=691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:36:43.942644971 (44m59.723578354s) [seg_pos={0 0}, upload=691]: Begin upload segment... Jul 26 22:36:43.942715586 (44m59.723648966s) [seg_pos={0 0}, upload=691]: Done begin upload segment. Jul 26 22:36:43.942752085 (44m59.723685471s) [noise=true , address=79.127.205.246:4287 , piece_id=AHNHIJ6LYZ2N7UZXOCLJP37IXFC7GFFWIZLJHVELORER5AUI27LA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.943373384 (44m59.724306777s) [noise=true , address=45.91.116.10:4100 , piece_id=XYDVDHRMMD5SYULA6TL2P2EEWLUQDGRVUQJ64OFIBG5QFYIIJRVA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.943838441 (44m59.72477183s) [noise=true , address=104.36.82.45:4936 , piece_id=LSFD24OKUPW3MA5XIEYAUUVM2IPISVT3R7IPSVKRCIRUSMT4JPJA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.944268010 (44m59.725201382s) [noise=true , address=193.149.164.143:4702 , piece_id=N3BGCB6E3VHAKMMNVKH4TDY3VE3LESIYFCLWPYIZIRWWCNVSYNVQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.944678500 (44m59.72561189s) [noise=true , address=79.127.219.47:4061 , piece_id=HVS44Q3QXKXNVJ3FRPVOKPV2ITJW5HIUUG7CT627LML6HJPNBMIQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945065556 (44m59.725998962s) [noise=true , address=104.36.82.34:4565 , piece_id=E5TVWQSLOW36QYLZJPX3B35AFSUWIKVQ7BKVPXTLW3TQ7CKZIWIQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945462411 (44m59.726395801s) [noise=true , address=79.127.219.51:4210 , piece_id=HRYS7V5THQGB2ICXY3Z3FRWBTZIEYLAAOYBVONHOKNIS7MGZCZXA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945856178 (44m59.726789567s) [noise=true , address=77.90.31.11:4558 , piece_id=X4EWCLG4OZDNRWSTL5JGBOWIIAYY7FHCZPXY7IILV4ZFOVQRXSVQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.946244188 (44m59.727177623s) [noise=true , address=79.127.205.245:4221 , piece_id=HASU6Q44AMAEJ3FMRYFMVMRNRWVWHXQ6S4FREG4BQ2ZADBS44CBQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.946660379 (44m59.727593771s) [noise=true , address=156.146.43.225:4448 , piece_id=R3VG6RUKSNVZUQTLDSN34CT3OCDE2YL22AN6EODCVK44P32LZKQQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.947097799 (44m59.728031189s) [noise=true , address=79.127.205.233:4656 , piece_id=HP4XRH5CWO7ZM7VREM7PGQB2VB7RG2ZPWSN33YDASOFMBXOZLTGQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.950911616 (44m59.731845003s) [noise=true , address=79.127.201.215:4341 , piece_id=6PUJV5VF37C4U7I2Z42HMUCJHHBI5KRD6RDMJMHX6KVVQFLSTFHQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.951551416 (44m59.732484802s) [noise=true , address=77.90.31.15:4732 , piece_id=KFS73H6RQZVH7RQDXRPUOS54FGQNW4PUUQEPQCL44G4H322Y2LGQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.952021135 (44m59.73295452s) [noise=true , address=79.127.219.38:4017 , piece_id=S5AFCQCIRJCFX734MBO6PB7C4O6A5JRILR5XQFPVG4SIM4AQJ5OQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.952615232 (44m59.733548621s) [noise=true , address=79.127.205.234:4618 , piece_id=KA4DCNR2M7HIZBRDZCO5JXJVR4HEJMJ5PWUOGHVDPN224Z6DSSKA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.953194884 (44m59.734128265s) [noise=true , address=77.90.31.18:4831 , piece_id=QS3CARJPIZIZCJQTHON6DHEQEKIFCYDCKCGIVFTE3AZDOAVAZISQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.953919127 (44m59.734852514s) [noise=true , address=64.49.13.170:4719 , piece_id=5XGJZB2F4RRZ57PFHNLPGXEAUOJRN3YD7HHKIHU65G3E6NVLGTKQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.954395925 (44m59.735329313s) [noise=true , address=79.127.205.235:4688 , piece_id=4GFOOOBMY6HN3Y7HQV72Y53X4RN34KHF6Z56CGJUX3CJIWRZ2NLQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:44.048957514 (44m59.829890894s) [upload=691]: Next returned nil segment Jul 26 22:36:44.048977726 (44m59.829911114s) [upload=691]: Waiting for error group managing segments... Jul 26 22:36:44.053653368 (44m59.834586742s) [noise=true , address=79.127.219.38:4017 , piece_id=S5AFCQCIRJCFX734MBO6PB7C4O6A5JRILR5XQFPVG4SIM4AQJ5OQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.076905665 (44m59.857839032s) [noise=true , address=77.90.31.11:4558 , piece_id=X4EWCLG4OZDNRWSTL5JGBOWIIAYY7FHCZPXY7IILV4ZFOVQRXSVQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.112521058 (44m59.893454439s) [noise=true , address=156.146.43.225:4448 , piece_id=R3VG6RUKSNVZUQTLDSN34CT3OCDE2YL22AN6EODCVK44P32LZKQQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:44.550215284 (45m0.331148671s) [noise=true , address=79.127.205.234:4618 , piece_id=KA4DCNR2M7HIZBRDZCO5JXJVR4HEJMJ5PWUOGHVDPN224Z6DSSKA, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.566685558 (45m0.347618929s) [noise=true , address=79.127.205.245:4221 , piece_id=HASU6Q44AMAEJ3FMRYFMVMRNRWVWHXQ6S4FREG4BQ2ZADBS44CBQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.583061209 (45m0.363994591s) [noise=true , address=79.127.205.233:4656 , piece_id=HP4XRH5CWO7ZM7VREM7PGQB2VB7RG2ZPWSN33YDASOFMBXOZLTGQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.612532419 (45m0.393465802s) [noise=true , address=79.127.219.51:4210 , piece_id=HRYS7V5THQGB2ICXY3Z3FRWBTZIEYLAAOYBVONHOKNIS7MGZCZXA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:44.647749987 (45m0.428683349s) [noise=true , address=79.127.205.246:4287 , piece_id=AHNHIJ6LYZ2N7UZXOCLJP37IXFC7GFFWIZLJHVELORER5AUI27LA, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.764033389 (45m0.544966768s) [noise=true , address=79.127.205.235:4688 , piece_id=4GFOOOBMY6HN3Y7HQV72Y53X4RN34KHF6Z56CGJUX3CJIWRZ2NLQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:45.359283634 (45m1.140217014s) [noise=true , address=45.91.116.10:4100 , piece_id=XYDVDHRMMD5SYULA6TL2P2EEWLUQDGRVUQJ64OFIBG5QFYIIJRVA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:46.520984499 (45m2.301917869s) [noise=true , address=79.127.201.215:4341 , piece_id=6PUJV5VF37C4U7I2Z42HMUCJHHBI5KRD6RDMJMHX6KVVQFLSTFHQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:46.832078832 (45m2.613012203s) [noise=true , address=79.127.219.47:4061 , piece_id=HVS44Q3QXKXNVJ3FRPVOKPV2ITJW5HIUUG7CT627LML6HJPNBMIQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:47.033893712 (45m2.814827102s) [noise=true , address=193.149.164.143:4702 , piece_id=N3BGCB6E3VHAKMMNVKH4TDY3VE3LESIYFCLWPYIZIRWWCNVSYNVQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:48.145235648 (45m3.926169033s) [noise=true , address=64.49.13.170:4719 , piece_id=5XGJZB2F4RRZ57PFHNLPGXEAUOJRN3YD7HHKIHU65G3E6NVLGTKQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:37:43.869125014 (45m59.65005838s) [noise=true , address=104.36.82.45:4936 , piece_id=LSFD24OKUPW3MA5XIEYAUUVM2IPISVT3R7IPSVKRCIRUSMT4JPJA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8, address:104.36.82.45:4936): protocol: write tcp 159.223.143.39:57056->104.36.82.45:4936: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869315595 (45m59.65024897s) [noise=true , address=77.90.31.15:4732 , piece_id=KFS73H6RQZVH7RQDXRPUOS54FGQNW4PUUQEPQCL44G4H322Y2LGQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: write tcp 159.223.143.39:40916->77.90.31.15:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869502930 (45m59.650436305s) [noise=true , address=77.90.31.18:4831 , piece_id=QS3CARJPIZIZCJQTHON6DHEQEKIFCYDCKCGIVFTE3AZDOAVAZISQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: write tcp 159.223.143.39:58740->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869645094 (45m59.650578461s) [noise=true , address=104.36.82.34:4565 , piece_id=E5TVWQSLOW36QYLZJPX3B35AFSUWIKVQ7BKVPXTLW3TQ7CKZIWIQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1JRxcRCxwuNJrRRcKKxEggFpmRk2fo9kdVbmhPn3eLLeiQop9R, address:104.36.82.34:4565): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:37:43.869697588 (45m59.650630973s) [seg_pos={0 0}, upload=691]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:37:43.904972836 (45m59.685906213s) [upload=691]: Done uploading segments... Jul 26 22:37:43.905001091 (45m59.685934467s) [upload=691]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 15.484107057s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 629.949881ms FAIL (2025-07-26 22:37:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 630.062255ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...............................X........X.X.X...XX..........] duration: 9.359107455s FAIL (2025-07-26 22:37:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.359864808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [X........XXX.XXX.X.XX..X..X......X.............X............] duration: 644.579124ms FAIL (2025-07-26 22:37:46 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 644.595811ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX] duration: 1m0.037908954s FAIL (2025-07-26 22:37:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 22:36:43.901527855 (44m59.682461222s) [upload=691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:36:43.901542041 (44m59.682475408s) [upload=691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:36:43.942644971 (44m59.723578354s) [seg_pos={0 0}, upload=691]: Begin upload segment... Jul 26 22:36:43.942715586 (44m59.723648966s) [seg_pos={0 0}, upload=691]: Done begin upload segment. Jul 26 22:36:43.942752085 (44m59.723685471s) [noise=true , address=79.127.205.246:4287 , piece_id=AHNHIJ6LYZ2N7UZXOCLJP37IXFC7GFFWIZLJHVELORER5AUI27LA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.943373384 (44m59.724306777s) [noise=true , address=45.91.116.10:4100 , piece_id=XYDVDHRMMD5SYULA6TL2P2EEWLUQDGRVUQJ64OFIBG5QFYIIJRVA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.943838441 (44m59.72477183s) [noise=true , address=104.36.82.45:4936 , piece_id=LSFD24OKUPW3MA5XIEYAUUVM2IPISVT3R7IPSVKRCIRUSMT4JPJA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.944268010 (44m59.725201382s) [noise=true , address=193.149.164.143:4702 , piece_id=N3BGCB6E3VHAKMMNVKH4TDY3VE3LESIYFCLWPYIZIRWWCNVSYNVQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.944678500 (44m59.72561189s) [noise=true , address=79.127.219.47:4061 , piece_id=HVS44Q3QXKXNVJ3FRPVOKPV2ITJW5HIUUG7CT627LML6HJPNBMIQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945065556 (44m59.725998962s) [noise=true , address=104.36.82.34:4565 , piece_id=E5TVWQSLOW36QYLZJPX3B35AFSUWIKVQ7BKVPXTLW3TQ7CKZIWIQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945462411 (44m59.726395801s) [noise=true , address=79.127.219.51:4210 , piece_id=HRYS7V5THQGB2ICXY3Z3FRWBTZIEYLAAOYBVONHOKNIS7MGZCZXA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.945856178 (44m59.726789567s) [noise=true , address=77.90.31.11:4558 , piece_id=X4EWCLG4OZDNRWSTL5JGBOWIIAYY7FHCZPXY7IILV4ZFOVQRXSVQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.946244188 (44m59.727177623s) [noise=true , address=79.127.205.245:4221 , piece_id=HASU6Q44AMAEJ3FMRYFMVMRNRWVWHXQ6S4FREG4BQ2ZADBS44CBQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.946660379 (44m59.727593771s) [noise=true , address=156.146.43.225:4448 , piece_id=R3VG6RUKSNVZUQTLDSN34CT3OCDE2YL22AN6EODCVK44P32LZKQQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.947097799 (44m59.728031189s) [noise=true , address=79.127.205.233:4656 , piece_id=HP4XRH5CWO7ZM7VREM7PGQB2VB7RG2ZPWSN33YDASOFMBXOZLTGQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.950911616 (44m59.731845003s) [noise=true , address=79.127.201.215:4341 , piece_id=6PUJV5VF37C4U7I2Z42HMUCJHHBI5KRD6RDMJMHX6KVVQFLSTFHQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.951551416 (44m59.732484802s) [noise=true , address=77.90.31.15:4732 , piece_id=KFS73H6RQZVH7RQDXRPUOS54FGQNW4PUUQEPQCL44G4H322Y2LGQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.952021135 (44m59.73295452s) [noise=true , address=79.127.219.38:4017 , piece_id=S5AFCQCIRJCFX734MBO6PB7C4O6A5JRILR5XQFPVG4SIM4AQJ5OQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.952615232 (44m59.733548621s) [noise=true , address=79.127.205.234:4618 , piece_id=KA4DCNR2M7HIZBRDZCO5JXJVR4HEJMJ5PWUOGHVDPN224Z6DSSKA, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.953194884 (44m59.734128265s) [noise=true , address=77.90.31.18:4831 , piece_id=QS3CARJPIZIZCJQTHON6DHEQEKIFCYDCKCGIVFTE3AZDOAVAZISQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.953919127 (44m59.734852514s) [noise=true , address=64.49.13.170:4719 , piece_id=5XGJZB2F4RRZ57PFHNLPGXEAUOJRN3YD7HHKIHU65G3E6NVLGTKQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:43.954395925 (44m59.735329313s) [noise=true , address=79.127.205.235:4688 , piece_id=4GFOOOBMY6HN3Y7HQV72Y53X4RN34KHF6Z56CGJUX3CJIWRZ2NLQ, seg_pos={0 0}, upload=691]: Uploading piece... Jul 26 22:36:44.048957514 (44m59.829890894s) [upload=691]: Next returned nil segment Jul 26 22:36:44.048977726 (44m59.829911114s) [upload=691]: Waiting for error group managing segments... Jul 26 22:36:44.053653368 (44m59.834586742s) [noise=true , address=79.127.219.38:4017 , piece_id=S5AFCQCIRJCFX734MBO6PB7C4O6A5JRILR5XQFPVG4SIM4AQJ5OQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.076905665 (44m59.857839032s) [noise=true , address=77.90.31.11:4558 , piece_id=X4EWCLG4OZDNRWSTL5JGBOWIIAYY7FHCZPXY7IILV4ZFOVQRXSVQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.112521058 (44m59.893454439s) [noise=true , address=156.146.43.225:4448 , piece_id=R3VG6RUKSNVZUQTLDSN34CT3OCDE2YL22AN6EODCVK44P32LZKQQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:44.550215284 (45m0.331148671s) [noise=true , address=79.127.205.234:4618 , piece_id=KA4DCNR2M7HIZBRDZCO5JXJVR4HEJMJ5PWUOGHVDPN224Z6DSSKA, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.566685558 (45m0.347618929s) [noise=true , address=79.127.205.245:4221 , piece_id=HASU6Q44AMAEJ3FMRYFMVMRNRWVWHXQ6S4FREG4BQ2ZADBS44CBQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.583061209 (45m0.363994591s) [noise=true , address=79.127.205.233:4656 , piece_id=HP4XRH5CWO7ZM7VREM7PGQB2VB7RG2ZPWSN33YDASOFMBXOZLTGQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.612532419 (45m0.393465802s) [noise=true , address=79.127.219.51:4210 , piece_id=HRYS7V5THQGB2ICXY3Z3FRWBTZIEYLAAOYBVONHOKNIS7MGZCZXA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:44.647749987 (45m0.428683349s) [noise=true , address=79.127.205.246:4287 , piece_id=AHNHIJ6LYZ2N7UZXOCLJP37IXFC7GFFWIZLJHVELORER5AUI27LA, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:44.764033389 (45m0.544966768s) [noise=true , address=79.127.205.235:4688 , piece_id=4GFOOOBMY6HN3Y7HQV72Y53X4RN34KHF6Z56CGJUX3CJIWRZ2NLQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:45.359283634 (45m1.140217014s) [noise=true , address=45.91.116.10:4100 , piece_id=XYDVDHRMMD5SYULA6TL2P2EEWLUQDGRVUQJ64OFIBG5QFYIIJRVA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:46.520984499 (45m2.301917869s) [noise=true , address=79.127.201.215:4341 , piece_id=6PUJV5VF37C4U7I2Z42HMUCJHHBI5KRD6RDMJMHX6KVVQFLSTFHQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:46.832078832 (45m2.613012203s) [noise=true , address=79.127.219.47:4061 , piece_id=HVS44Q3QXKXNVJ3FRPVOKPV2ITJW5HIUUG7CT627LML6HJPNBMIQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:36:47.033893712 (45m2.814827102s) [noise=true , address=193.149.164.143:4702 , piece_id=N3BGCB6E3VHAKMMNVKH4TDY3VE3LESIYFCLWPYIZIRWWCNVSYNVQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:48.145235648 (45m3.926169033s) [noise=true , address=64.49.13.170:4719 , piece_id=5XGJZB2F4RRZ57PFHNLPGXEAUOJRN3YD7HHKIHU65G3E6NVLGTKQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: <nil> Jul 26 22:37:43.869125014 (45m59.65005838s) [noise=true , address=104.36.82.45:4936 , piece_id=LSFD24OKUPW3MA5XIEYAUUVM2IPISVT3R7IPSVKRCIRUSMT4JPJA, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8, address:104.36.82.45:4936): protocol: write tcp 159.223.143.39:57056->104.36.82.45:4936: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869315595 (45m59.65024897s) [noise=true , address=77.90.31.15:4732 , piece_id=KFS73H6RQZVH7RQDXRPUOS54FGQNW4PUUQEPQCL44G4H322Y2LGQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: write tcp 159.223.143.39:40916->77.90.31.15:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869502930 (45m59.650436305s) [noise=true , address=77.90.31.18:4831 , piece_id=QS3CARJPIZIZCJQTHON6DHEQEKIFCYDCKCGIVFTE3AZDOAVAZISQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: write tcp 159.223.143.39:58740->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:37:43.869645094 (45m59.650578461s) [noise=true , address=104.36.82.34:4565 , piece_id=E5TVWQSLOW36QYLZJPX3B35AFSUWIKVQ7BKVPXTLW3TQ7CKZIWIQ, seg_pos={0 0}, upload=691]: Done uploading piece. err: ecclient: upload failed (node:1JRxcRCxwuNJrRRcKKxEggFpmRk2fo9kdVbmhPn3eLLeiQop9R, address:104.36.82.34:4565): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:37:43.869697588 (45m59.650630973s) [seg_pos={0 0}, upload=691]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:37:43.904972836 (45m59.685906213s) [upload=691]: Done uploading segments... Jul 26 22:37:43.905001091 (45m59.685934467s) [upload=691]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.037938429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 242.71597ms FAIL (2025-07-26 22:37:24 UTC) orbiter warning: uplink: too many requests Duration: 242.76039ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [................................X........X.X.X...XX.........] duration: 23.497625938s FAIL (2025-07-26 22:37:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.497646183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....XX..XXX..X..XX..X..XX.X....X....X........X.............] duration: 498.386449ms FAIL (2025-07-26 22:37:05 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 498.419584ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 499.170886ms FAIL (2025-07-26 22:36:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 499.197325ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......XX..XXX..X..XX..X..XX.X....X....X........X............] duration: 748.695551ms FAIL (2025-07-26 22:36:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 748.726315ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X........XXX.XXX.X.XX..X..X......X.............X...........] duration: 517.439325ms FAIL (2025-07-26 22:36:45 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 517.478145ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.................................X........X.X.X...XX........] duration: 43.772930232s FAIL (2025-07-26 22:36:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.77296209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 271.915213ms FAIL (2025-07-26 22:36:43 UTC) orbiter warning: uplink: too many requests Duration: 271.945406ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 40 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XX] duration: 1m0.034621651s FAIL (2025-07-26 22:36:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 6803691s) [upload=675]: Uploading segments... Jul 26 22:35:30.687181937 (43m46.46811533s) [upload=675]: Got next segment. Inline: false Jul 26 22:35:30.687233799 (43m46.468167169s) [upload=675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:35:30.687239765 (43m46.46817315s) [upload=675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:35:30.728984867 (43m46.509918258s) [seg_pos={0 0}, upload=675]: Begin upload segment... Jul 26 22:35:30.729061688 (43m46.509995069s) [seg_pos={0 0}, upload=675]: Done begin upload segment. Jul 26 22:35:30.729097186 (43m46.510030574s) [noise=true , address=79.127.219.47:4048 , piece_id=QRZOSWTJLWGVEAU22CQIVRAXL7Q36XXAKU5QUKEUC342HZCNSMVQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.729652133 (43m46.510585522s) [noise=true , address=79.127.220.101:4048 , piece_id=IUPOOQESSZ23AXCOSV7QBJ7575JUZXEORLSTXMSWHXBTBD7ZZG2Q, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.730062086 (43m46.510995472s) [noise=true , address=79.127.219.38:4001 , piece_id=IHZPK256ADZE6FBYZ2SPXTZLDLVHF2QQYXH3UJ3XNABDDC4Q6NMA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.730451131 (43m46.511384519s) [noise=true , address=77.90.31.17:4787 , piece_id=LXM5IHAXGXEVSCEMBRZOLBLH6QW4KFNVNZ453PBO2BGTQ2UEKDIA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.730838247 (43m46.511771633s) [noise=true , address=79.127.219.51:4212 , piece_id=KR2627EFSL37ZN7DY2NK27COAEJN2SZPVE4VM733RJDWG4YWQHMQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.731256076 (43m46.512189465s) [noise=true , address=77.90.31.16:4743 , piece_id=BHEZR3DMDDJ4LMFKTYSWFY2PJOGY4QBG3N2R43UAPYEANZSD7XLQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.731686837 (43m46.512620226s) [noise=true , address=77.90.31.18:4831 , piece_id=AZM43A6FZCVL57DNBZNYTV6H52XZNRHGLUULM2SXOVF3KEQ4USWQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.732423866 (43m46.513357255s) [noise=true , address=77.90.31.19:4867 , piece_id=LNNOECSGUO4GYMOQPFMLX3SXDHKQCLAXKPEQNDWVDWVKZ5S4VOQA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.732872501 (43m46.513805888s) [noise=true , address=79.127.219.48:4090 , piece_id=Q64O75TAH4CS2OGAOMWVR76RHCQNF2AVVLOB7OHDCBJ2FMS2R4UQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.733254781 (43m46.51418817s) [noise=true , address=79.127.205.246:4285 , piece_id=CHRRUNCJDF4HSPPBLJLKSZDR4XOJPQCWAPVWBZWH2E7DZXH7DXWQ, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.733664752 (43m46.51459813s) [noise=true , address=79.127.205.234:4601 , piece_id=AISC6QJ44WOHECCE7OVBV7X6WRRVX4EXESCHTJR3IEBGMUC7AL3Q, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.734139169 (43m46.515072536s) [noise=true , address=64.49.13.170:4721 , piece_id=RXL6GSXCBSTSPFRAEY3O73XGXXVDFGCLJEPDKPLJ3RJU3JGTES2A, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.734553484 (43m46.515486876s) [noise=true , address=45.91.116.9:4269 , piece_id=VNR7SCF6KMHXULRFZKSZV7A7VHCEX2LN4YBFQHCOWQ7GO5X3IW4Q, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.734963101 (43m46.51589649s) [noise=true , address=104.36.82.11:4138 , piece_id=PVYONMLKHQX6HGCM45ZTN2TAJMXOAM6PNURSBPRHNRF7FTUQI4TA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.735421554 (43m46.516354943s) [noise=true , address=79.127.201.215:4359 , piece_id=ZNSGRGVJWNKHZPQG5BXNY7M6NBOX7J34BOFLJVLKFNPPSEOUKY4Q, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.735828597 (43m46.516761985s) [noise=true , address=77.90.31.15:4723 , piece_id=XT32Y4DX554I3KVOCKS6NTIP3ASCQSAUPLV5NXG66WEGH7TENTNA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.736270428 (43m46.517203815s) [noise=true , address=64.49.13.138:4482 , piece_id=LRNWJIBYUO4LEXHXBOB6EDY7QVTLQHCBWF4Q4CL5PXTKSFT7WDIA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.736674151 (43m46.517607543s) [noise=true , address=156.146.43.225:4448 , piece_id=YEHPZKKHQLJ57VOK6RENN4PFNJ6QYHN26Z3T47C4D6EM67Y4OOKA, seg_pos={0 0}, upload=675]: Uploading piece... Jul 26 22:35:30.840412650 (43m46.621346033s) [upload=675]: Next returned nil segment Jul 26 22:35:30.840434059 (43m46.621367444s) [upload=675]: Waiting for error group managing segments... Jul 26 22:35:30.887100731 (43m46.668034109s) [noise=true , address=77.90.31.17:4787 , piece_id=LXM5IHAXGXEVSCEMBRZOLBLH6QW4KFNVNZ453PBO2BGTQ2UEKDIA, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:30.974047219 (43m46.754980597s) [noise=true , address=79.127.219.38:4001 , piece_id=IHZPK256ADZE6FBYZ2SPXTZLDLVHF2QQYXH3UJ3XNABDDC4Q6NMA, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:31.012374471 (43m46.793307855s) [noise=true , address=79.127.219.51:4212 , piece_id=KR2627EFSL37ZN7DY2NK27COAEJN2SZPVE4VM733RJDWG4YWQHMQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:35:31.349945905 (43m47.13087928s) [noise=true , address=79.127.205.234:4601 , piece_id=AISC6QJ44WOHECCE7OVBV7X6WRRVX4EXESCHTJR3IEBGMUC7AL3Q, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:32.028901166 (43m47.809834536s) [noise=true , address=79.127.219.48:4090 , piece_id=Q64O75TAH4CS2OGAOMWVR76RHCQNF2AVVLOB7OHDCBJ2FMS2R4UQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:32.175720135 (43m47.956653511s) [noise=true , address=45.91.116.9:4269 , piece_id=VNR7SCF6KMHXULRFZKSZV7A7VHCEX2LN4YBFQHCOWQ7GO5X3IW4Q, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: storage node overloaded, request limit: 2000 Jul 26 22:35:32.314503227 (43m48.095436606s) [noise=true , address=79.127.220.101:4048 , piece_id=IUPOOQESSZ23AXCOSV7QBJ7575JUZXEORLSTXMSWHXBTBD7ZZG2Q, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:32.344389872 (43m48.125323238s) [noise=true , address=156.146.43.225:4448 , piece_id=YEHPZKKHQLJ57VOK6RENN4PFNJ6QYHN26Z3T47C4D6EM67Y4OOKA, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:33.777908892 (43m49.558842265s) [noise=true , address=79.127.219.47:4048 , piece_id=QRZOSWTJLWGVEAU22CQIVRAXL7Q36XXAKU5QUKEUC342HZCNSMVQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:36.839672883 (43m52.620606249s) [noise=true , address=64.49.13.138:4482 , piece_id=LRNWJIBYUO4LEXHXBOB6EDY7QVTLQHCBWF4Q4CL5PXTKSFT7WDIA, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:37.287385531 (43m53.068318901s) [noise=true , address=64.49.13.170:4721 , piece_id=RXL6GSXCBSTSPFRAEY3O73XGXXVDFGCLJEPDKPLJ3RJU3JGTES2A, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:37.761222869 (43m53.542156234s) [noise=true , address=79.127.205.246:4285 , piece_id=CHRRUNCJDF4HSPPBLJLKSZDR4XOJPQCWAPVWBZWH2E7DZXH7DXWQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:35:41.191286987 (43m56.972220365s) [noise=true , address=79.127.201.215:4359 , piece_id=ZNSGRGVJWNKHZPQG5BXNY7M6NBOX7J34BOFLJVLKFNPPSEOUKY4Q, seg_pos={0 0}, upload=675]: Done uploading piece. err: <nil> Jul 26 22:36:26.095801764 (44m41.876735148s) [noise=true , address=77.90.31.16:4743 , piece_id=BHEZR3DMDDJ4LMFKTYSWFY2PJOGY4QBG3N2R43UAPYEANZSD7XLQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: storage node overloaded, request limit: 2000 Jul 26 22:36:30.656359404 (44m46.437292775s) [noise=true , address=77.90.31.18:4831 , piece_id=AZM43A6FZCVL57DNBZNYTV6H52XZNRHGLUULM2SXOVF3KEQ4USWQ, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: rpc: dial tcp 77.90.31.18:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:36:30.656440720 (44m46.4373741s) [noise=true , address=77.90.31.15:4723 , piece_id=XT32Y4DX554I3KVOCKS6NTIP3ASCQSAUPLV5NXG66WEGH7TENTNA, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: rpc: dial tcp 77.90.31.15:4723: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:36:30.656479718 (44m46.437413097s) [noise=true , address=77.90.31.19:4867 , piece_id=LNNOECSGUO4GYMOQPFMLX3SXDHKQCLAXKPEQNDWVDWVKZ5S4VOQA, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: write tcp 159.223.143.39:60320->77.90.31.19:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:36:30.656518955 (44m46.437452334s) [noise=true , address=104.36.82.11:4138 , piece_id=PVYONMLKHQX6HGCM45ZTN2TAJMXOAM6PNURSBPRHNRF7FTUQI4TA, seg_pos={0 0}, upload=675]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: write tcp 159.223.143.39:42174->104.36.82.11:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:36:30.656571521 (44m46.43750491s) [seg_pos={0 0}, upload=675]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:36:30.686366243 (44m46.467299612s) [upload=675]: Done uploading segments... Jul 26 22:36:30.686414382 (44m46.467347759s) [upload=675]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.034665738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X........XXX.XXX.X.XX..X..X......X.............X..........] duration: 523.605603ms FAIL (2025-07-26 22:36:19 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 523.638438ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..................................X........X.X.X...XX.......] duration: 19.814772684s FAIL (2025-07-26 22:36:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.814813635s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 497.843122ms FAIL (2025-07-26 22:35:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 497.881438ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......XX..XXX..X..XX..X..XX.X....X....X........X...........] duration: 608.243877ms FAIL (2025-07-26 22:35:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 608.370789ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...................................X........X.X.X...XX......] duration: 19.370492542s FAIL (2025-07-26 22:35:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.370515065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 215.329677ms FAIL (2025-07-26 22:35:38 UTC) orbiter warning: uplink: too many requests Duration: 215.367725ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..X] duration: 1m0.041840454s FAIL (2025-07-26 22:35:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 661]: Got next segment. Inline: false Jul 26 22:34:27.951601133 (42m43.732534511s) [upload=661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:34:27.951620077 (42m43.732553458s) [upload=661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:34:27.987616768 (42m43.768550155s) [seg_pos={0 0}, upload=661]: Begin upload segment... Jul 26 22:34:27.987677035 (42m43.768610417s) [seg_pos={0 0}, upload=661]: Done begin upload segment. Jul 26 22:34:27.987710731 (42m43.76864416s) [noise=true , address=64.49.13.140:4516 , piece_id=VNEH2NXYNH4HC6UNLLIA4VI56ORP57CGDIVPR7BFONOLDIY7H6PA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.988347840 (42m43.769281227s) [noise=true , address=79.127.205.246:4288 , piece_id=VHIKAX2FZBGPSFRZMAWF6J27NHRBL6TSHE5FZMET6DXKXLWUUWUA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.988766163 (42m43.769699552s) [noise=true , address=79.127.219.51:4212 , piece_id=LYUAPOLRDGBWIDGMDX6LI4X2LZZP2QHOV2TOBEZHEGXKNZUZW3HA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.989132146 (42m43.770065535s) [noise=true , address=45.91.116.13:4175 , piece_id=ZLRYM2YSOEZ5NWSOWK5BPKYQIANMVJVSGP37DT64E4VUO7P6HTDQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.989479830 (42m43.770413219s) [noise=true , address=79.127.201.215:4347 , piece_id=XCKRUQYGDAUMHA22VBVP64UCZISA2GDA5PYW6LGFCB7I6577KXUQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.989836216 (42m43.770769605s) [noise=true , address=77.90.31.19:4871 , piece_id=QVCO3O6M37CI6BJOSOPAIAMIAUFXB2RGZNYPK3TBAZEADAKETM2A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.990228021 (42m43.77116141s) [noise=true , address=79.127.219.47:4069 , piece_id=4THURBUEKRP3JM7QZCWVRE5Y5ILP3GI26LA7X7QODSSJ4PNYY6EA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.993010821 (42m43.773944208s) [noise=true , address=77.90.31.17:4774 , piece_id=QG6LKYVX23N5L4CQFZ6IMRVOTLACEZUIOTBFOAZSLJFTEVLSRDIQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.993550663 (42m43.774484051s) [noise=true , address=77.90.31.15:4713 , piece_id=7ONT4B7UNK53T4LYXCAXK6NBUQGN45Z27DYZZYU4AI4QHDKRXS3A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.994855210 (42m43.7757886s) [noise=true , address=79.127.220.97:4361 , piece_id=2FH5LIZV2S5RAHNKOUOZMOGXA5VVGYIADMKQFTHSDYLFOVJNAD2Q, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.995342767 (42m43.776276154s) [noise=true , address=45.91.116.9:4267 , piece_id=2VHHVANAMTK6KINVQVVEPIUUTKJL3KJABZROESEE2SOLTMSBL3RQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.995778002 (42m43.776711391s) [noise=true , address=64.49.13.138:4482 , piece_id=NM5Z2MIKYTIP7XUEKOGUM4WVBYY5ZJZDMTSW4YV2BNHYA7VB6R7A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.996318448 (42m43.777251835s) [noise=true , address=79.127.219.38:4003 , piece_id=T3HRQHU4G34QQPLHXBXW5IMJTFKO2BWS4WSWRJKGO3M6D3FSW7ZA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.996711652 (42m43.777645038s) [noise=true , address=79.127.205.234:4612 , piece_id=PCVIO3LMYYN4TINAB5DVUS47NCHCRKHQOMQWMQIH26H4SEJQSC5A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.997100832 (42m43.778034226s) [noise=true , address=79.127.205.245:4237 , piece_id=GEJFAX5OPXASZESLJLRPOVE5W6LLUFHDP2KIO7FK4U37KY5T6FIA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.997489534 (42m43.778422924s) [noise=true , address=45.91.116.8:4065 , piece_id=X255M7QFAT5JOPBZO3DYBNHOKDKSSBODKMKXF7ZKVDSW7EW7TGHQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.997921939 (42m43.77885533s) [noise=true , address=77.90.31.12:4608 , piece_id=QCZE7AFBCTBJCNAVR67KQSU3FRNVNQQQTFP3LQ4QQRHBIKWCXJLQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.998345471 (42m43.77927886s) [noise=true , address=77.90.31.18:4817 , piece_id=NSAQ4EDSLCYZSSN7TB4HEQ5XXVVH6QSPHXJKIDI36BITOYYMVXXA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:28.056033124 (42m43.836966506s) [noise=true , address=79.127.219.47:4069 , piece_id=4THURBUEKRP3JM7QZCWVRE5Y5ILP3GI26LA7X7QODSSJ4PNYY6EA, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:18dK6RxbMxU4a46pW4YPiQW87xgpeoTb1ouqE9fzTdzTAFAa3W, address:79.127.219.47:4069): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:28.095674502 (42m43.876607888s) [upload=661]: Next returned nil segment Jul 26 22:34:28.095713720 (42m43.876647086s) [upload=661]: Waiting for error group managing segments... Jul 26 22:34:28.101217341 (42m43.882150718s) [noise=true , address=79.127.219.38:4003 , piece_id=T3HRQHU4G34QQPLHXBXW5IMJTFKO2BWS4WSWRJKGO3M6D3FSW7ZA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.119364618 (42m43.900298008s) [noise=true , address=79.127.219.51:4212 , piece_id=LYUAPOLRDGBWIDGMDX6LI4X2LZZP2QHOV2TOBEZHEGXKNZUZW3HA, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:28.138620488 (42m43.919553877s) [noise=true , address=79.127.220.97:4361 , piece_id=2FH5LIZV2S5RAHNKOUOZMOGXA5VVGYIADMKQFTHSDYLFOVJNAD2Q, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:28.144505433 (42m43.92543881s) [noise=true , address=77.90.31.17:4774 , piece_id=QG6LKYVX23N5L4CQFZ6IMRVOTLACEZUIOTBFOAZSLJFTEVLSRDIQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.146709992 (42m43.927643371s) [noise=true , address=77.90.31.12:4608 , piece_id=QCZE7AFBCTBJCNAVR67KQSU3FRNVNQQQTFP3LQ4QQRHBIKWCXJLQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.631921429 (42m44.412854796s) [noise=true , address=79.127.205.246:4288 , piece_id=VHIKAX2FZBGPSFRZMAWF6J27NHRBL6TSHE5FZMET6DXKXLWUUWUA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.678668075 (42m44.459601453s) [noise=true , address=79.127.205.234:4612 , piece_id=PCVIO3LMYYN4TINAB5DVUS47NCHCRKHQOMQWMQIH26H4SEJQSC5A, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:29.137248803 (42m44.918182166s) [noise=true , address=79.127.205.245:4237 , piece_id=GEJFAX5OPXASZESLJLRPOVE5W6LLUFHDP2KIO7FK4U37KY5T6FIA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:31.771346822 (42m47.552280203s) [noise=true , address=45.91.116.13:4175 , piece_id=ZLRYM2YSOEZ5NWSOWK5BPKYQIANMVJVSGP37DT64E4VUO7P6HTDQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:32.565641205 (42m48.346574594s) [noise=true , address=45.91.116.8:4065 , piece_id=X255M7QFAT5JOPBZO3DYBNHOKDKSSBODKMKXF7ZKVDSW7EW7TGHQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:17EPCQmYb816aegE9vPSoJYByMUEvhP4DUiabis2gh9QjXQC3s, address:45.91.116.8:4065): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:33.360957537 (42m49.141890915s) [noise=true , address=79.127.201.215:4347 , piece_id=XCKRUQYGDAUMHA22VBVP64UCZISA2GDA5PYW6LGFCB7I6577KXUQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:34.784741480 (42m50.565674868s) [noise=true , address=45.91.116.9:4267 , piece_id=2VHHVANAMTK6KINVQVVEPIUUTKJL3KJABZROESEE2SOLTMSBL3RQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:35.843967454 (42m51.624900832s) [noise=true , address=64.49.13.138:4482 , piece_id=NM5Z2MIKYTIP7XUEKOGUM4WVBYY5ZJZDMTSW4YV2BNHYA7VB6R7A, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:36.075665804 (42m51.85659917s) [noise=true , address=64.49.13.140:4516 , piece_id=VNEH2NXYNH4HC6UNLLIA4VI56ORP57CGDIVPR7BFONOLDIY7H6PA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:35:27.909515279 (43m43.690448663s) [noise=true , address=77.90.31.19:4871 , piece_id=QVCO3O6M37CI6BJOSOPAIAMIAUFXB2RGZNYPK3TBAZEADAKETM2A, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:35:27.909647421 (43m43.690580801s) [noise=true , address=77.90.31.15:4713 , piece_id=7ONT4B7UNK53T4LYXCAXK6NBUQGN45Z27DYZZYU4AI4QHDKRXS3A, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: write tcp 159.223.143.39:53460->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:35:27.909740858 (43m43.690674235s) [noise=true , address=77.90.31.18:4817 , piece_id=NSAQ4EDSLCYZSSN7TB4HEQ5XXVVH6QSPHXJKIDI36BITOYYMVXXA, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: write tcp 159.223.143.39:56042->77.90.31.18:4817: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:35:27.909776921 (43m43.690710298s) [seg_pos={0 0}, upload=661]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:35:27.947311027 (43m43.728244408s) [upload=661]: Done uploading segments... Jul 26 22:35:27.947342152 (43m43.72827553s) [upload=661]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.041877536s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 489.097538ms FAIL (2025-07-26 22:35:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 489.131552ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [X..X........XXX.XXX.X.XX..X..X......X.............X.........] duration: 441.152948ms FAIL (2025-07-26 22:35:16 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 441.184713ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 357.16577ms FAIL (2025-07-26 22:34:53 UTC) orbiter warning: uplink: too many requests Duration: 357.196019ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [........XX..XXX..X..XX..X..XX.X....X....X........X..........] duration: 417.635535ms FAIL (2025-07-26 22:34:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.78023ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 522.967641ms FAIL (2025-07-26 22:34:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 522.990313ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....................................X........X.X.X...XX.....] duration: 1.53728997s FAIL (2025-07-26 22:34:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.538325747s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 750.789941ms FAIL (2025-07-26 22:34:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 750.847867ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..X........XXX.XXX.X.XX..X..X......X.............X........] duration: 493.470179ms FAIL (2025-07-26 22:34:12 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 493.499066ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........XX..XXX..X..XX..X..XX.X....X....X........X.........] duration: 717.26462ms FAIL (2025-07-26 22:34:09 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 717.30003ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....................................X........X.X.X...XX....] duration: 1.940973544s FAIL (2025-07-26 22:34:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.94101523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..] duration: 33.715051763s FAIL (2025-07-26 22:33:54 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0200 (39m38.953613576s) [upload=602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:31:25.092817481 (39m40.873750866s) [upload=615]: Starting upload Jul 26 22:31:25.092917455 (39m40.873850841s) [upload=615]: Done starting upload Jul 26 22:31:25.093095667 (39m40.874029048s) [upload=615]: Uploading segments... Jul 26 22:31:25.093538635 (39m40.874472036s) [upload=615]: Got next segment. Inline: false Jul 26 22:31:25.093618508 (39m40.874551888s) [upload=615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:31:25.093624913 (39m40.874558297s) [upload=615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:31:25.131355162 (39m40.912288549s) [seg_pos={0 0}, upload=615]: Begin upload segment... Jul 26 22:31:25.131427766 (39m40.912361148s) [seg_pos={0 0}, upload=615]: Done begin upload segment. Jul 26 22:31:25.131463629 (39m40.912397017s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.131879895 (39m40.91281328s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132543852 (39m40.91347724s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132579299 (39m40.913512688s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133014796 (39m40.913948163s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133072582 (39m40.914005971s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133510863 (39m40.914444253s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133529028 (39m40.914462418s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133983589 (39m40.914917048s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134054538 (39m40.914987927s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134498895 (39m40.915432285s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134590489 (39m40.915523878s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134966793 (39m40.915900183s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135163729 (39m40.916097118s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135677391 (39m40.916610776s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135741108 (39m40.916674484s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.136284624 (39m40.917218003s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.137524409 (39m40.918457794s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.240969311 (39m41.021902693s) [upload=615]: Next returned nil segment Jul 26 22:31:25.240995734 (39m41.021929111s) [upload=615]: Waiting for error group managing segments... Jul 26 22:31:25.243994267 (39m41.024927648s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12bQXjK4bv5583RV4BtjhC8WTDJgz255Ptvs6f5kqvJu4SA99mc, address:79.127.220.101:4072): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.245375223 (39m41.026308585s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.291788207 (39m41.07272157s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.388770962 (39m41.169704331s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.516260375 (39m41.297193757s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.723538202 (39m41.504471568s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.725133974 (39m41.506067342s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.748427597 (39m41.529360974s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.786472821 (39m41.567406198s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:26.151056459 (39m41.931989836s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.431220045 (39m42.212153426s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.952353227 (39m42.733286605s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:27.155824153 (39m42.936757545s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:27.445015172 (39m43.225948552s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:34.971697059 (39m50.752630428s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:42.955261339 (39m58.73619492s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:51.785580862 (40m7.566514236s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:32:25.059574940 (40m40.840508328s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:16EACrmREmTxBUPQKwsABtESUoZKWzcxKCknE81xK1nrDYpZSg, address:77.90.31.18:4822): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:32:25.059680674 (40m40.840614041s) [seg_pos={0 0}, upload=615]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:32:25.088124602 (40m40.869057975s) [upload=615]: Done uploading segments... Jul 26 22:32:25.088153768 (40m40.869087129s) [upload=615]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 33.715086785s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 456.542025ms FAIL (2025-07-26 22:33:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.562389ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 209.313242ms FAIL (2025-07-26 22:33:48 UTC) orbiter warning: uplink: too many requests Duration: 209.34397ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......................................X........X.X.X...XX...] duration: 17.184905353s FAIL (2025-07-26 22:33:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.184943029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X..X........XXX.XXX.X.XX..X..X......X.............X.......] duration: 447.304014ms FAIL (2025-07-26 22:33:01 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 447.345575ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..........XX..XXX..X..XX..X..XX.X....X....X........X........] duration: 434.62897ms FAIL (2025-07-26 22:32:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.656449ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X..X........XXX.XXX.X.XX..X..X......X.............X......] duration: 435.00656ms FAIL (2025-07-26 22:32:46 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 435.034962ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 546.386073ms FAIL (2025-07-26 22:32:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 546.415953ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 207.121511ms FAIL (2025-07-26 22:32:37 UTC) orbiter warning: uplink: too many requests Duration: 207.155143ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.] duration: 5.73014305s FAIL (2025-07-26 22:32:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0200 (39m38.953613576s) [upload=602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:31:25.092817481 (39m40.873750866s) [upload=615]: Starting upload Jul 26 22:31:25.092917455 (39m40.873850841s) [upload=615]: Done starting upload Jul 26 22:31:25.093095667 (39m40.874029048s) [upload=615]: Uploading segments... Jul 26 22:31:25.093538635 (39m40.874472036s) [upload=615]: Got next segment. Inline: false Jul 26 22:31:25.093618508 (39m40.874551888s) [upload=615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:31:25.093624913 (39m40.874558297s) [upload=615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:31:25.131355162 (39m40.912288549s) [seg_pos={0 0}, upload=615]: Begin upload segment... Jul 26 22:31:25.131427766 (39m40.912361148s) [seg_pos={0 0}, upload=615]: Done begin upload segment. Jul 26 22:31:25.131463629 (39m40.912397017s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.131879895 (39m40.91281328s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132543852 (39m40.91347724s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132579299 (39m40.913512688s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133014796 (39m40.913948163s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133072582 (39m40.914005971s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133510863 (39m40.914444253s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133529028 (39m40.914462418s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133983589 (39m40.914917048s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134054538 (39m40.914987927s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134498895 (39m40.915432285s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134590489 (39m40.915523878s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134966793 (39m40.915900183s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135163729 (39m40.916097118s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135677391 (39m40.916610776s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135741108 (39m40.916674484s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.136284624 (39m40.917218003s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.137524409 (39m40.918457794s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.240969311 (39m41.021902693s) [upload=615]: Next returned nil segment Jul 26 22:31:25.240995734 (39m41.021929111s) [upload=615]: Waiting for error group managing segments... Jul 26 22:31:25.243994267 (39m41.024927648s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12bQXjK4bv5583RV4BtjhC8WTDJgz255Ptvs6f5kqvJu4SA99mc, address:79.127.220.101:4072): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.245375223 (39m41.026308585s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.291788207 (39m41.07272157s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.388770962 (39m41.169704331s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.516260375 (39m41.297193757s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.723538202 (39m41.504471568s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.725133974 (39m41.506067342s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.748427597 (39m41.529360974s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.786472821 (39m41.567406198s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:26.151056459 (39m41.931989836s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.431220045 (39m42.212153426s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.952353227 (39m42.733286605s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:27.155824153 (39m42.936757545s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:27.445015172 (39m43.225948552s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:34.971697059 (39m50.752630428s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:42.955261339 (39m58.73619492s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:51.785580862 (40m7.566514236s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:32:25.059574940 (40m40.840508328s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:16EACrmREmTxBUPQKwsABtESUoZKWzcxKCknE81xK1nrDYpZSg, address:77.90.31.18:4822): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:32:25.059680674 (40m40.840614041s) [seg_pos={0 0}, upload=615]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:32:25.088124602 (40m40.869057975s) [upload=615]: Done uploading segments... Jul 26 22:32:25.088153768 (40m40.869087129s) [upload=615]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.730167129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......................................X........X.X.X...XX..] duration: 4.834799903s FAIL (2025-07-26 22:32:27 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.834834902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration: 1m0.030676676s FAIL (2025-07-26 22:32:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0200 (39m38.953613576s) [upload=602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:31:25.092817481 (39m40.873750866s) [upload=615]: Starting upload Jul 26 22:31:25.092917455 (39m40.873850841s) [upload=615]: Done starting upload Jul 26 22:31:25.093095667 (39m40.874029048s) [upload=615]: Uploading segments... Jul 26 22:31:25.093538635 (39m40.874472036s) [upload=615]: Got next segment. Inline: false Jul 26 22:31:25.093618508 (39m40.874551888s) [upload=615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:31:25.093624913 (39m40.874558297s) [upload=615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:31:25.131355162 (39m40.912288549s) [seg_pos={0 0}, upload=615]: Begin upload segment... Jul 26 22:31:25.131427766 (39m40.912361148s) [seg_pos={0 0}, upload=615]: Done begin upload segment. Jul 26 22:31:25.131463629 (39m40.912397017s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.131879895 (39m40.91281328s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132543852 (39m40.91347724s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132579299 (39m40.913512688s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133014796 (39m40.913948163s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133072582 (39m40.914005971s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133510863 (39m40.914444253s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133529028 (39m40.914462418s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133983589 (39m40.914917048s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134054538 (39m40.914987927s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134498895 (39m40.915432285s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134590489 (39m40.915523878s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134966793 (39m40.915900183s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135163729 (39m40.916097118s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135677391 (39m40.916610776s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135741108 (39m40.916674484s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.136284624 (39m40.917218003s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.137524409 (39m40.918457794s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.240969311 (39m41.021902693s) [upload=615]: Next returned nil segment Jul 26 22:31:25.240995734 (39m41.021929111s) [upload=615]: Waiting for error group managing segments... Jul 26 22:31:25.243994267 (39m41.024927648s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12bQXjK4bv5583RV4BtjhC8WTDJgz255Ptvs6f5kqvJu4SA99mc, address:79.127.220.101:4072): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.245375223 (39m41.026308585s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.291788207 (39m41.07272157s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.388770962 (39m41.169704331s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.516260375 (39m41.297193757s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.723538202 (39m41.504471568s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.725133974 (39m41.506067342s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.748427597 (39m41.529360974s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.786472821 (39m41.567406198s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:26.151056459 (39m41.931989836s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.431220045 (39m42.212153426s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.952353227 (39m42.733286605s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:27.155824153 (39m42.936757545s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:27.445015172 (39m43.225948552s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:34.971697059 (39m50.752630428s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:42.955261339 (39m58.73619492s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:51.785580862 (40m7.566514236s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:32:25.059574940 (40m40.840508328s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:16EACrmREmTxBUPQKwsABtESUoZKWzcxKCknE81xK1nrDYpZSg, address:77.90.31.18:4822): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:32:25.059680674 (40m40.840614041s) [seg_pos={0 0}, upload=615]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:32:25.088124602 (40m40.869057975s) [upload=615]: Done uploading segments... Jul 26 22:32:25.088153768 (40m40.869087129s) [upload=615]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.03072904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 247.309138ms FAIL (2025-07-26 22:32:05 UTC) orbiter warning: uplink: too many requests Duration: 247.346203ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...........XX..XXX..X..XX..X..XX.X....X....X........X.......] duration: 641.280149ms FAIL (2025-07-26 22:31:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 641.315695ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X..X........XXX.XXX.X.XX..X..X......X.............X.....] duration: 541.603125ms FAIL (2025-07-26 22:31:54 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 541.63824ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 587.336207ms FAIL (2025-07-26 22:31:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 587.418487ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 1m0.029674709s FAIL (2025-07-26 22:31:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:30:08.761600623 (38m24.542534006s) [upload=582]: Done uploading segments... Jul 26 22:30:08.761635235 (38m24.542568618s) [upload=582]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Jul 26 22:30:23.175715785 (38m38.956649175s) [upload=602]: Starting upload Jul 26 22:30:23.176643641 (38m38.957577037s) [upload=602]: Done starting upload Jul 26 22:30:23.176858032 (38m38.957791403s) [upload=602]: Uploading segments... Jul 26 22:30:23.177442532 (38m38.958375909s) [upload=602]: Got next segment. Inline: false Jul 26 22:30:23.177529730 (38m38.958463096s) [upload=602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:30:23.177539149 (38m38.958472517s) [upload=602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:30:23.220426626 (38m39.001360014s) [seg_pos={0 0}, upload=602]: Begin upload segment... Jul 26 22:30:23.220496207 (38m39.001429595s) [seg_pos={0 0}, upload=602]: Done begin upload segment. Jul 26 22:30:23.220541622 (38m39.001475014s) [noise=true , address=79.127.201.215:4360 , piece_id=LDXCX2G3H2CUYF42CODPHX5A3DBFEVEGHR7NTBVUK2T6MIZAEM3Q, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.221481322 (38m39.002414711s) [noise=true , address=77.90.31.17:4796 , piece_id=LNOIFXAW622A3JSV3NXD3NFRSGYMPZ4ZYR62P3P6CJND4BJMEEIA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.222101054 (38m39.003034443s) [noise=true , address=79.127.205.234:4613 , piece_id=JRJH33VQB7YJTIGMXDKUZ2Z6JWD7S5WIUNJDJJFURPVA3W7IMZYA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.222767744 (38m39.003701132s) [noise=true , address=77.90.31.11:4580 , piece_id=UXKISZP65MHRIMHI3VILH7OAD3HJUQMEQ25EZCU6NZDYJF75QS7A, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.223371716 (38m39.004305103s) [noise=true , address=45.91.116.8:4065 , piece_id=ZWC5LB4UE3MN4FLC67IJUPR2OQCPM3YVFZTBL4IR2EZYRX363VFA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.223841406 (38m39.00477479s) [noise=true , address=45.91.116.14:4206 , piece_id=ZHFDNVYDBGHR4TAV4NYDQG7DCJ5YS6O7P6EWM6O5KH2RLJVE3P4A, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.224429753 (38m39.005363139s) [noise=true , address=77.90.31.21:4909 , piece_id=7BDNPKYYTBHFT34IUIXXKN7DM77M54PFKP7CUGWXTU3MMQUAMI3Q, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.224883320 (38m39.005816708s) [noise=true , address=45.91.116.5:4310 , piece_id=BX6SKOJY27LBLEIS3OKXUG2ISJBKBR2TEMBUBA6AQYWM3SGUHYVQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.225319227 (38m39.006252616s) [noise=true , address=64.49.13.169:4716 , piece_id=DISVDESQ35BEY3YDILQLFPKBI6KPNZTBKZSC4N2BFOHB6UATZM7Q, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.226710795 (38m39.007644183s) [noise=true , address=79.127.205.233:4652 , piece_id=PJUN5Q7AAGAIYBVFL2XVK7HOHN2VSUYYFMJFA2KAYUVKBR3GH2SQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.227512914 (38m39.00844641s) [noise=true , address=79.127.219.51:4201 , piece_id=7UOX3EMEC2TJKOD5TH5OQK7IGACKSDKS3427NS6LIACG2P3DGUDQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.227953319 (38m39.008886708s) [noise=true , address=64.49.13.170:4721 , piece_id=HJ233QGSCIENIKGWK6OC4FO6IHIICJY26LBG7ABUWQYHNDRL3JPQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.228555621 (38m39.00948901s) [noise=true , address=45.91.116.12:4229 , piece_id=HHVTYQRHUQHYJPZX3D3LH54QD545CE7VKALR2UL37IIVKIOLIPPA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.229094745 (38m39.010028132s) [noise=true , address=77.90.31.13:4651 , piece_id=5U273HIPJOE2ZPGFT6V5QMIVYMQGZ4AA4HMGLMVFEDVT7SXRGBQA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.229653467 (38m39.010586846s) [noise=true , address=79.127.205.245:4243 , piece_id=W6B3YDPNH2KGKQ3Q5UNMRV6JHFQZIPFVXPR6EXB6PIDUUVTMEOZA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.230260750 (38m39.011194139s) [noise=true , address=64.49.13.138:4483 , piece_id=X2TXDPN3IWXTA5WLA2QKPRCPW2S6SIQBWJ5E7I4DCEZA4C52YEPQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.230834024 (38m39.011767412s) [noise=true , address=77.90.31.15:4714 , piece_id=RCQ7JYIFAZ24CO7T5XFZWP3MK3QOFB57TX5PNQLKHIUDNGDU6MGQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.231466514 (38m39.012399898s) [noise=true , address=79.127.205.235:4682 , piece_id=NIUEICIFB67KGMLQVXEMSMPM32UKW622IPGF4F6RAW54IXIZMDMA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.358902366 (38m39.139835729s) [upload=602]: Next returned nil segment Jul 26 22:30:23.358921206 (38m39.139854582s) [upload=602]: Waiting for error group managing segments... Jul 26 22:30:23.396711622 (38m39.177644997s) [noise=true , address=77.90.31.17:4796 , piece_id=LNOIFXAW622A3JSV3NXD3NFRSGYMPZ4ZYR62P3P6CJND4BJMEEIA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.429135315 (38m39.210068678s) [noise=true , address=77.90.31.13:4651 , piece_id=5U273HIPJOE2ZPGFT6V5QMIVYMQGZ4AA4HMGLMVFEDVT7SXRGBQA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.492792568 (38m39.273725937s) [noise=true , address=77.90.31.11:4580 , piece_id=UXKISZP65MHRIMHI3VILH7OAD3HJUQMEQ25EZCU6NZDYJF75QS7A, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.786085237 (38m39.567018615s) [noise=true , address=64.49.13.170:4721 , piece_id=HJ233QGSCIENIKGWK6OC4FO6IHIICJY26LBG7ABUWQYHNDRL3JPQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:23.833627956 (38m39.614561334s) [noise=true , address=79.127.205.233:4652 , piece_id=PJUN5Q7AAGAIYBVFL2XVK7HOHN2VSUYYFMJFA2KAYUVKBR3GH2SQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.850607137 (38m39.631540513s) [noise=true , address=79.127.205.245:4243 , piece_id=W6B3YDPNH2KGKQ3Q5UNMRV6JHFQZIPFVXPR6EXB6PIDUUVTMEOZA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.963675075 (38m39.744608451s) [noise=true , address=45.91.116.8:4065 , piece_id=ZWC5LB4UE3MN4FLC67IJUPR2OQCPM3YVFZTBL4IR2EZYRX363VFA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:24.037010439 (38m39.817943809s) [noise=true , address=79.127.205.234:4613 , piece_id=JRJH33VQB7YJTIGMXDKUZ2Z6JWD7S5WIUNJDJJFURPVA3W7IMZYA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:24.192023890 (38m39.972957259s) [noise=true , address=79.127.205.235:4682 , piece_id=NIUEICIFB67KGMLQVXEMSMPM32UKW622IPGF4F6RAW54IXIZMDMA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:24.994113890 (38m40.775047251s) [noise=true , address=79.127.219.51:4201 , piece_id=7UOX3EMEC2TJKOD5TH5OQK7IGACKSDKS3427NS6LIACG2P3DGUDQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:26.699149799 (38m42.480083167s) [noise=true , address=64.49.13.169:4716 , piece_id=DISVDESQ35BEY3YDILQLFPKBI6KPNZTBKZSC4N2BFOHB6UATZM7Q, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:28.523850606 (38m44.304783973s) [noise=true , address=45.91.116.14:4206 , piece_id=ZHFDNVYDBGHR4TAV4NYDQG7DCJ5YS6O7P6EWM6O5KH2RLJVE3P4A, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:28.684871960 (38m44.465805324s) [noise=true , address=45.91.116.12:4229 , piece_id=HHVTYQRHUQHYJPZX3D3LH54QD545CE7VKALR2UL37IIVKIOLIPPA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:31.099634536 (38m46.880567918s) [noise=true , address=45.91.116.5:4310 , piece_id=BX6SKOJY27LBLEIS3OKXUG2ISJBKBR2TEMBUBA6AQYWM3SGUHYVQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:31.341563839 (38m47.122497218s) [noise=true , address=64.49.13.138:4483 , piece_id=X2TXDPN3IWXTA5WLA2QKPRCPW2S6SIQBWJ5E7I4DCEZA4C52YEPQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:54.205996972 (39m9.986930349s) [noise=true , address=79.127.201.215:4360 , piece_id=LDXCX2G3H2CUYF42CODPHX5A3DBFEVEGHR7NTBVUK2T6MIZAEM3Q, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:31:23.146066980 (39m38.927000363s) [noise=true , address=77.90.31.15:4714 , piece_id=RCQ7JYIFAZ24CO7T5XFZWP3MK3QOFB57TX5PNQLKHIUDNGDU6MGQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: rpc: dial tcp 77.90.31.15:4714: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:31:23.146140712 (39m38.927074091s) [noise=true , address=77.90.31.21:4909 , piece_id=7BDNPKYYTBHFT34IUIXXKN7DM77M54PFKP7CUGWXTU3MMQUAMI3Q, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:45828->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:31:23.146194450 (39m38.927127835s) [seg_pos={0 0}, upload=602]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:31:23.172654781 (39m38.953588151s) [upload=602]: Done uploading segments... Jul 26 22:31:23.172680200 (39m38.953613576s) [upload=602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.029725569s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)