Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..X..XX..X..XX.X....X....X........X........................] duration: 637.358141ms FAIL (2025-07-26 22:45:09 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 637.390894ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXX.XXX.X.XX..X..X......X.............X................X....] duration: 375.612162ms FAIL (2025-07-26 22:44:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 375.639533ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.] duration: 6.264211583s FAIL (2025-07-26 22:44:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 26 22:43:41.110295795 (51m56.891229164s) [upload=788]: Done uploading segments... Jul 26 22:43:41.110322252 (51m56.891255627s) [upload=788]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:43:41.194853734 (51m56.975787099s) [upload=802]: Starting upload Jul 26 22:43:41.194948018 (51m56.975881403s) [upload=802]: Done starting upload Jul 26 22:43:41.195214873 (51m56.97614824s) [upload=802]: Uploading segments... Jul 26 22:43:41.195440638 (51m56.976374016s) [upload=802]: Got next segment. Inline: false Jul 26 22:43:41.195523504 (51m56.976456868s) [upload=802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:41.195529671 (51m56.976463035s) [upload=802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:41.231947842 (51m57.012881229s) [seg_pos={0 0}, upload=802]: Begin upload segment... Jul 26 22:43:41.232018620 (51m57.012952006s) [seg_pos={0 0}, upload=802]: Done begin upload segment. Jul 26 22:43:41.232103172 (51m57.013036547s) [noise=true , address=77.90.31.19:4874 , piece_id=RLVLQGUYAJRH53SBRIA25EE6L3YCXCHIN7CQ22MGFE5QJZECJYNQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232134530 (51m57.013067913s) [noise=true , address=79.127.205.245:4228 , piece_id=IIVPND2UAM7VD6DA4HGKOX3B3ZVHNYP63ZPWNMN3F47UVZ57TD6A, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232598891 (51m57.013532278s) [noise=true , address=104.36.82.35:4607 , piece_id=X36Y42PQULA5A4B6UZMCPQK34FM4INNMUMQHKKZL2FG6MOELOJFA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232997403 (51m57.013930793s) [noise=true , address=64.49.13.138:4482 , piece_id=HTUBWJUH7EUYOMFWBLQX25WAJGRJCH3DGX2V6HAZSJ2IUMEI3LCQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.233811231 (51m57.014744615s) [noise=true , address=64.49.13.137:4500 , piece_id=QJKNHVY5IRULGIELVRDNXQMWI4C6XLPLKH5QX3OLEG4IEOIN4MZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234107016 (51m57.0150404s) [noise=true , address=79.127.219.48:4090 , piece_id=SMYUBZNSBP4OWQZFEIZ5GIZ6QRYLEKXJI5EFDKHSZJID744JTTDA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234225439 (51m57.015158817s) [noise=true , address=79.127.219.51:4210 , piece_id=N5X4TBNI5MBOJKBING7YL55LVQU6B5ZQBSGAM3YQFTVACO4BSC5Q, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234530128 (51m57.01546354s) [noise=true , address=104.36.82.10:4101 , piece_id=S6RT7PW6TVY7K2KNB42YKXEQMDTSMOFJK4CDNO3MKB6FDXLVYBXA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234677395 (51m57.015610776s) [noise=true , address=77.90.31.18:4816 , piece_id=HY55JOE2FS3B3EGYJ3C43MTAXQ33EBQKUUX27SPJ4IMRFVDDOOZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235151053 (51m57.016084431s) [noise=true , address=79.127.220.101:4048 , piece_id=UUV32R7XFPCNDEF43ZOMWC67JJZ7U7YDMLXFXAHTIXUKESNCB4MA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235529599 (51m57.016462976s) [noise=true , address=156.146.43.225:4449 , piece_id=G44WXWCQDQOSSOBG64LRAU3DLOUSH2ULL4I3GT6THFAGYD6BQRTA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235990344 (51m57.016923728s) [noise=true , address=79.127.219.38:4004 , piece_id=JPDFRQ2YEH3KF6RGT4QIE47C6ZI6HMHBTUOXTHGRYZ4ZNDMFEZJQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.236414748 (51m57.017348125s) [noise=true , address=79.127.205.246:4282 , piece_id=HC27CTDAG7TRJ5YQQFFMG7SFDRIN3JRTMFUI42KMK35T3TIB4TZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.237577245 (51m57.018510629s) [noise=true , address=104.36.82.7:4918 , piece_id=3KJZTIAZBP7WBZMHYTYYO2XWUBANMLBT7KUCUKTYHSJ2MZRKXQAA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.238217941 (51m57.019151322s) [noise=true , address=79.127.205.233:4633 , piece_id=VZE6UZ4SW2XBZRNZMBP4TTDDVBRQGYM6SQKYR5BPYBV446K5KPGA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.238950587 (51m57.019883971s) [noise=true , address=77.90.31.11:4571 , piece_id=5XMT4IU4WYZNCLUHFKE2ZYDJTGIX6ESHWWHTNOW22IUB3GE5V3JQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.239460727 (51m57.020394109s) [noise=true , address=77.90.31.10:4538 , piece_id=Z6GPO6B4JMM27B5CJE64GZMAEDWAJKAAVJBRCPRAB5ULHMI7XLFQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.239899361 (51m57.020832748s) [noise=true , address=79.127.205.234:4616 , piece_id=XQSAFSV4ZC5ACSFVVWKAL7BPLQ2KVT2ZEYU6LUM3Q3EFOP3S22PQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.304182241 (51m57.085115671s) [upload=802]: Next returned nil segment Jul 26 22:43:41.304206569 (51m57.085139943s) [upload=802]: Waiting for error group managing segments... Jul 26 22:43:41.309312769 (51m57.090246201s) [noise=true , address=79.127.219.38:4004 , piece_id=JPDFRQ2YEH3KF6RGT4QIE47C6ZI6HMHBTUOXTHGRYZ4ZNDMFEZJQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.366028377 (51m57.146961744s) [noise=true , address=77.90.31.11:4571 , piece_id=5XMT4IU4WYZNCLUHFKE2ZYDJTGIX6ESHWWHTNOW22IUB3GE5V3JQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.385684427 (51m57.166617806s) [noise=true , address=79.127.219.51:4210 , piece_id=N5X4TBNI5MBOJKBING7YL55LVQU6B5ZQBSGAM3YQFTVACO4BSC5Q, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:41.823755988 (51m57.604689367s) [noise=true , address=79.127.205.246:4282 , piece_id=HC27CTDAG7TRJ5YQQFFMG7SFDRIN3JRTMFUI42KMK35T3TIB4TZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.837645590 (51m57.61857896s) [noise=true , address=79.127.205.245:4228 , piece_id=IIVPND2UAM7VD6DA4HGKOX3B3ZVHNYP63ZPWNMN3F47UVZ57TD6A, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.944332236 (51m57.725265612s) [noise=true , address=64.49.13.138:4482 , piece_id=HTUBWJUH7EUYOMFWBLQX25WAJGRJCH3DGX2V6HAZSJ2IUMEI3LCQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:42.082424997 (51m57.86335838s) [noise=true , address=79.127.205.234:4616 , piece_id=XQSAFSV4ZC5ACSFVVWKAL7BPLQ2KVT2ZEYU6LUM3Q3EFOP3S22PQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:42.339902702 (51m58.120836064s) [noise=true , address=79.127.205.233:4633 , piece_id=VZE6UZ4SW2XBZRNZMBP4TTDDVBRQGYM6SQKYR5BPYBV446K5KPGA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:42.616942028 (51m58.397875405s) [noise=true , address=79.127.219.48:4090 , piece_id=SMYUBZNSBP4OWQZFEIZ5GIZ6QRYLEKXJI5EFDKHSZJID744JTTDA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:43.761718655 (51m59.542652024s) [noise=true , address=156.146.43.225:4449 , piece_id=G44WXWCQDQOSSOBG64LRAU3DLOUSH2ULL4I3GT6THFAGYD6BQRTA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:44.144083824 (51m59.925017201s) [noise=true , address=64.49.13.137:4500 , piece_id=QJKNHVY5IRULGIELVRDNXQMWI4C6XLPLKH5QX3OLEG4IEOIN4MZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:45.274140250 (52m1.055073612s) [noise=true , address=79.127.220.101:4048 , piece_id=UUV32R7XFPCNDEF43ZOMWC67JJZ7U7YDMLXFXAHTIXUKESNCB4MA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:46.921015676 (52m2.70194904s) [noise=true , address=77.90.31.19:4874 , piece_id=RLVLQGUYAJRH53SBRIA25EE6L3YCXCHIN7CQ22MGFE5QJZECJYNQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:52.019885342 (52m7.800818718s) [noise=true , address=77.90.31.10:4538 , piece_id=Z6GPO6B4JMM27B5CJE64GZMAEDWAJKAAVJBRCPRAB5ULHMI7XLFQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:59.877267403 (52m15.658200785s) [noise=true , address=77.90.31.18:4816 , piece_id=HY55JOE2FS3B3EGYJ3C43MTAXQ33EBQKUUX27SPJ4IMRFVDDOOZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:44:41.166057950 (52m56.946991332s) [noise=true , address=104.36.82.35:4607 , piece_id=X36Y42PQULA5A4B6UZMCPQK34FM4INNMUMQHKKZL2FG6MOELOJFA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1Ex8eUvL1Hqxd5EffckSvBeZDpGrMGhKENyCPvqHybtQf3YQzM, address:104.36.82.35:4607): protocol: rpc: dial tcp 104.36.82.35:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166171721 (52m56.9471051s) [noise=true , address=104.36.82.10:4101 , piece_id=S6RT7PW6TVY7K2KNB42YKXEQMDTSMOFJK4CDNO3MKB6FDXLVYBXA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166252938 (52m56.947186315s) [noise=true , address=104.36.82.7:4918 , piece_id=3KJZTIAZBP7WBZMHYTYYO2XWUBANMLBT7KUCUKTYHSJ2MZRKXQAA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:12EMGZ4hpXmBp4TwemLrbKAVtjBmtexMzEahRTzFw42qQipsAG, address:104.36.82.7:4918): protocol: write tcp 159.223.143.39:37030->104.36.82.7:4918: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166278974 (52m56.947212365s) [seg_pos={0 0}, upload=802]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:44:41.189117481 (52m56.97005085s) [upload=802]: Done uploading segments... Jul 26 22:44:41.189141942 (52m56.970075317s) [upload=802]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.264232796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX] duration: 1m0.025681074s FAIL (2025-07-26 22:44:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Jul 26 22:43:41.110295795 (51m56.891229164s) [upload=788]: Done uploading segments... Jul 26 22:43:41.110322252 (51m56.891255627s) [upload=788]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:43:41.194853734 (51m56.975787099s) [upload=802]: Starting upload Jul 26 22:43:41.194948018 (51m56.975881403s) [upload=802]: Done starting upload Jul 26 22:43:41.195214873 (51m56.97614824s) [upload=802]: Uploading segments... Jul 26 22:43:41.195440638 (51m56.976374016s) [upload=802]: Got next segment. Inline: false Jul 26 22:43:41.195523504 (51m56.976456868s) [upload=802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:41.195529671 (51m56.976463035s) [upload=802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:41.231947842 (51m57.012881229s) [seg_pos={0 0}, upload=802]: Begin upload segment... Jul 26 22:43:41.232018620 (51m57.012952006s) [seg_pos={0 0}, upload=802]: Done begin upload segment. Jul 26 22:43:41.232103172 (51m57.013036547s) [noise=true , address=77.90.31.19:4874 , piece_id=RLVLQGUYAJRH53SBRIA25EE6L3YCXCHIN7CQ22MGFE5QJZECJYNQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232134530 (51m57.013067913s) [noise=true , address=79.127.205.245:4228 , piece_id=IIVPND2UAM7VD6DA4HGKOX3B3ZVHNYP63ZPWNMN3F47UVZ57TD6A, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232598891 (51m57.013532278s) [noise=true , address=104.36.82.35:4607 , piece_id=X36Y42PQULA5A4B6UZMCPQK34FM4INNMUMQHKKZL2FG6MOELOJFA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.232997403 (51m57.013930793s) [noise=true , address=64.49.13.138:4482 , piece_id=HTUBWJUH7EUYOMFWBLQX25WAJGRJCH3DGX2V6HAZSJ2IUMEI3LCQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.233811231 (51m57.014744615s) [noise=true , address=64.49.13.137:4500 , piece_id=QJKNHVY5IRULGIELVRDNXQMWI4C6XLPLKH5QX3OLEG4IEOIN4MZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234107016 (51m57.0150404s) [noise=true , address=79.127.219.48:4090 , piece_id=SMYUBZNSBP4OWQZFEIZ5GIZ6QRYLEKXJI5EFDKHSZJID744JTTDA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234225439 (51m57.015158817s) [noise=true , address=79.127.219.51:4210 , piece_id=N5X4TBNI5MBOJKBING7YL55LVQU6B5ZQBSGAM3YQFTVACO4BSC5Q, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234530128 (51m57.01546354s) [noise=true , address=104.36.82.10:4101 , piece_id=S6RT7PW6TVY7K2KNB42YKXEQMDTSMOFJK4CDNO3MKB6FDXLVYBXA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.234677395 (51m57.015610776s) [noise=true , address=77.90.31.18:4816 , piece_id=HY55JOE2FS3B3EGYJ3C43MTAXQ33EBQKUUX27SPJ4IMRFVDDOOZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235151053 (51m57.016084431s) [noise=true , address=79.127.220.101:4048 , piece_id=UUV32R7XFPCNDEF43ZOMWC67JJZ7U7YDMLXFXAHTIXUKESNCB4MA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235529599 (51m57.016462976s) [noise=true , address=156.146.43.225:4449 , piece_id=G44WXWCQDQOSSOBG64LRAU3DLOUSH2ULL4I3GT6THFAGYD6BQRTA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.235990344 (51m57.016923728s) [noise=true , address=79.127.219.38:4004 , piece_id=JPDFRQ2YEH3KF6RGT4QIE47C6ZI6HMHBTUOXTHGRYZ4ZNDMFEZJQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.236414748 (51m57.017348125s) [noise=true , address=79.127.205.246:4282 , piece_id=HC27CTDAG7TRJ5YQQFFMG7SFDRIN3JRTMFUI42KMK35T3TIB4TZQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.237577245 (51m57.018510629s) [noise=true , address=104.36.82.7:4918 , piece_id=3KJZTIAZBP7WBZMHYTYYO2XWUBANMLBT7KUCUKTYHSJ2MZRKXQAA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.238217941 (51m57.019151322s) [noise=true , address=79.127.205.233:4633 , piece_id=VZE6UZ4SW2XBZRNZMBP4TTDDVBRQGYM6SQKYR5BPYBV446K5KPGA, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.238950587 (51m57.019883971s) [noise=true , address=77.90.31.11:4571 , piece_id=5XMT4IU4WYZNCLUHFKE2ZYDJTGIX6ESHWWHTNOW22IUB3GE5V3JQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.239460727 (51m57.020394109s) [noise=true , address=77.90.31.10:4538 , piece_id=Z6GPO6B4JMM27B5CJE64GZMAEDWAJKAAVJBRCPRAB5ULHMI7XLFQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.239899361 (51m57.020832748s) [noise=true , address=79.127.205.234:4616 , piece_id=XQSAFSV4ZC5ACSFVVWKAL7BPLQ2KVT2ZEYU6LUM3Q3EFOP3S22PQ, seg_pos={0 0}, upload=802]: Uploading piece... Jul 26 22:43:41.304182241 (51m57.085115671s) [upload=802]: Next returned nil segment Jul 26 22:43:41.304206569 (51m57.085139943s) [upload=802]: Waiting for error group managing segments... Jul 26 22:43:41.309312769 (51m57.090246201s) [noise=true , address=79.127.219.38:4004 , piece_id=JPDFRQ2YEH3KF6RGT4QIE47C6ZI6HMHBTUOXTHGRYZ4ZNDMFEZJQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.366028377 (51m57.146961744s) [noise=true , address=77.90.31.11:4571 , piece_id=5XMT4IU4WYZNCLUHFKE2ZYDJTGIX6ESHWWHTNOW22IUB3GE5V3JQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.385684427 (51m57.166617806s) [noise=true , address=79.127.219.51:4210 , piece_id=N5X4TBNI5MBOJKBING7YL55LVQU6B5ZQBSGAM3YQFTVACO4BSC5Q, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:41.823755988 (51m57.604689367s) [noise=true , address=79.127.205.246:4282 , piece_id=HC27CTDAG7TRJ5YQQFFMG7SFDRIN3JRTMFUI42KMK35T3TIB4TZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.837645590 (51m57.61857896s) [noise=true , address=79.127.205.245:4228 , piece_id=IIVPND2UAM7VD6DA4HGKOX3B3ZVHNYP63ZPWNMN3F47UVZ57TD6A, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:41.944332236 (51m57.725265612s) [noise=true , address=64.49.13.138:4482 , piece_id=HTUBWJUH7EUYOMFWBLQX25WAJGRJCH3DGX2V6HAZSJ2IUMEI3LCQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:42.082424997 (51m57.86335838s) [noise=true , address=79.127.205.234:4616 , piece_id=XQSAFSV4ZC5ACSFVVWKAL7BPLQ2KVT2ZEYU6LUM3Q3EFOP3S22PQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:42.339902702 (51m58.120836064s) [noise=true , address=79.127.205.233:4633 , piece_id=VZE6UZ4SW2XBZRNZMBP4TTDDVBRQGYM6SQKYR5BPYBV446K5KPGA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:42.616942028 (51m58.397875405s) [noise=true , address=79.127.219.48:4090 , piece_id=SMYUBZNSBP4OWQZFEIZ5GIZ6QRYLEKXJI5EFDKHSZJID744JTTDA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:43.761718655 (51m59.542652024s) [noise=true , address=156.146.43.225:4449 , piece_id=G44WXWCQDQOSSOBG64LRAU3DLOUSH2ULL4I3GT6THFAGYD6BQRTA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:44.144083824 (51m59.925017201s) [noise=true , address=64.49.13.137:4500 , piece_id=QJKNHVY5IRULGIELVRDNXQMWI4C6XLPLKH5QX3OLEG4IEOIN4MZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:45.274140250 (52m1.055073612s) [noise=true , address=79.127.220.101:4048 , piece_id=UUV32R7XFPCNDEF43ZOMWC67JJZ7U7YDMLXFXAHTIXUKESNCB4MA, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:46.921015676 (52m2.70194904s) [noise=true , address=77.90.31.19:4874 , piece_id=RLVLQGUYAJRH53SBRIA25EE6L3YCXCHIN7CQ22MGFE5QJZECJYNQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:52.019885342 (52m7.800818718s) [noise=true , address=77.90.31.10:4538 , piece_id=Z6GPO6B4JMM27B5CJE64GZMAEDWAJKAAVJBRCPRAB5ULHMI7XLFQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:43:59.877267403 (52m15.658200785s) [noise=true , address=77.90.31.18:4816 , piece_id=HY55JOE2FS3B3EGYJ3C43MTAXQ33EBQKUUX27SPJ4IMRFVDDOOZQ, seg_pos={0 0}, upload=802]: Done uploading piece. err: <nil> Jul 26 22:44:41.166057950 (52m56.946991332s) [noise=true , address=104.36.82.35:4607 , piece_id=X36Y42PQULA5A4B6UZMCPQK34FM4INNMUMQHKKZL2FG6MOELOJFA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1Ex8eUvL1Hqxd5EffckSvBeZDpGrMGhKENyCPvqHybtQf3YQzM, address:104.36.82.35:4607): protocol: rpc: dial tcp 104.36.82.35:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166171721 (52m56.9471051s) [noise=true , address=104.36.82.10:4101 , piece_id=S6RT7PW6TVY7K2KNB42YKXEQMDTSMOFJK4CDNO3MKB6FDXLVYBXA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166252938 (52m56.947186315s) [noise=true , address=104.36.82.7:4918 , piece_id=3KJZTIAZBP7WBZMHYTYYO2XWUBANMLBT7KUCUKTYHSJ2MZRKXQAA, seg_pos={0 0}, upload=802]: Done uploading piece. err: ecclient: upload failed (node:12EMGZ4hpXmBp4TwemLrbKAVtjBmtexMzEahRTzFw42qQipsAG, address:104.36.82.7:4918): protocol: write tcp 159.223.143.39:37030->104.36.82.7:4918: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:44:41.166278974 (52m56.947212365s) [seg_pos={0 0}, upload=802]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:44:41.189117481 (52m56.97005085s) [upload=802]: Done uploading segments... Jul 26 22:44:41.189141942 (52m56.970075317s) [upload=802]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.025697982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XXX.XXX.X.XX..X..X......X.............X................X...] duration: 467.917602ms FAIL (2025-07-26 22:44:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.959855ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 207.169283ms FAIL (2025-07-26 22:44:20 UTC) orbiter warning: uplink: too many requests Duration: 207.197664ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX..X..XX..X..XX.X....X....X........X.......................] duration: 403.258317ms FAIL (2025-07-26 22:44:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 403.287104ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.933267332s FAIL (2025-07-26 22:44:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.9332919s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 46 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XX] duration: 1m0.026927028s FAIL (2025-07-26 22:43:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== closed: op Jul 26 22:42:31.875494050 (50m47.656427425s) [seg_pos={0 0}, upload=770]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:42:31.897672595 (50m47.678605975s) [upload=770]: Done uploading segments... Jul 26 22:42:31.897692998 (50m47.678626381s) [upload=770]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Jul 26 22:42:41.119669262 (50m56.90060265s) [upload=788]: Starting upload Jul 26 22:42:41.119746547 (50m56.900679932s) [upload=788]: Done starting upload Jul 26 22:42:41.119970071 (50m56.900903449s) [upload=788]: Uploading segments... Jul 26 22:42:41.120491867 (50m56.901425243s) [upload=788]: Got next segment. Inline: false Jul 26 22:42:41.120566591 (50m56.901499972s) [upload=788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:42:41.120576258 (50m56.901509636s) [upload=788]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:42:41.159698708 (50m56.940632096s) [seg_pos={0 0}, upload=788]: Begin upload segment... Jul 26 22:42:41.159778821 (50m56.940712184s) [seg_pos={0 0}, upload=788]: Done begin upload segment. Jul 26 22:42:41.159819902 (50m56.940753288s) [noise=true , address=79.127.219.38:4004 , piece_id=5LNGIYND5433E5Y6RIXUN5SJ2KZBHAL4VKNM2BETGAJI4ZDRNZJA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.160357367 (50m56.941290743s) [noise=true , address=45.91.116.5:4310 , piece_id=24PKHLS7LIWT5FSSOK6GTAGJA2QT23FGZBTYKCFRRQZIAFUOLJRQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.160740977 (50m56.941674363s) [noise=true , address=77.90.31.21:4900 , piece_id=BPIJVCVJFO2VULVEZCVXKZZJYK4T3ZAQR5SDR2YUED3WVGIHNRBA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.161159365 (50m56.942092751s) [noise=true , address=79.127.219.48:4090 , piece_id=N4YY2MEHQ5GDG5IEUHRZUYCFZEFKWWB6Z746X2BZ3DT53NDECIHA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.161585900 (50m56.942519287s) [noise=true , address=77.90.31.11:4560 , piece_id=2G27HVSVC2D7VMTF7UEKQ46CQOVXOWFYKV6G7R6RQNRJUKHY5O6Q, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.161900536 (50m56.942833901s) [noise=true , address=45.91.116.11:4124 , piece_id=KK2HXQVYAS2WPWBPO3RDXSJH5AJMJIHI45EKAJ4JXMSMYGUOOXWA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.162174785 (50m56.943108149s) [noise=true , address=104.36.82.52:4091 , piece_id=OPAS4F56AZ6MTZG7IGHHHPT3EHFSZKMV6R4BMS4WI2YGBDWAFTHA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.162504222 (50m56.943437606s) [noise=true , address=77.90.31.15:4715 , piece_id=VTL7LRR2FDDRNJMNG5SZDCTLL6NAUAEII52JPV2KSXRDANZ3LWIQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.162815008 (50m56.943748392s) [noise=true , address=79.127.201.215:4347 , piece_id=2YL6SRDVNRLUVTY52NPSD45CSI6GWLRVGFWA3CJO5AOU2INO7EBQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.163231431 (50m56.944164795s) [noise=true , address=79.127.205.233:4649 , piece_id=7L55UVAVWQFODXIYC6QBFLLVDSD6RZBXVAGJ2KGKTBVMPLYRIXCQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.163553660 (50m56.944487037s) [noise=true , address=77.90.31.16:4743 , piece_id=FJ6A2JBU2QRMBBR3S3X6PODEXDDVQ4K534JZYDPAHK2TBVLOO2GA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.163902378 (50m56.944835765s) [noise=true , address=79.127.205.234:4603 , piece_id=AHNYCS34ZHS7OYZGUNE4GRA2UYBZOTSG6BWN37RROM2AFHKTGEXQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.164241933 (50m56.945175313s) [noise=true , address=79.127.220.101:4048 , piece_id=FXAMGR6YV5ROTM4XXV4HIXLSLB6IUJDF5NBABCLGES6IWI6RNJXQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.164543860 (50m56.945477247s) [noise=true , address=77.90.31.18:4809 , piece_id=ZOCAFE6L3S252W7D6DJG7AO2AJWQMB6VDQLGZ5DE2AU45XMAYBAA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.164898196 (50m56.945831571s) [noise=true , address=156.146.43.225:4448 , piece_id=NICLKW2I7FRRO74H2RTUDJNR2XCONV74OAJDEZX52WQPS4PG2VFQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.165214086 (50m56.946147453s) [noise=true , address=104.36.82.44:4877 , piece_id=45N5BVVXQNCYEQTWEY5UE26PMYX34EZTXAU4IZZTUTMUNR7FFYRQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.165481585 (50m56.94641495s) [noise=true , address=79.127.205.245:4243 , piece_id=DIWMJGHBI4X3UH6VZNFOBET3FC3GN5JZXMTEQKIXDDNG5HPA7MQA, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.165794133 (50m56.946727514s) [noise=true , address=45.91.116.10:4100 , piece_id=R2OMVYXOKMSAKXYLG36Z5VBRZQJ47ZXS5EXWOMBUT37JJVX4LMEQ, seg_pos={0 0}, upload=788]: Uploading piece... Jul 26 22:42:41.242046962 (50m57.022980346s) [noise=true , address=77.90.31.18:4809 , piece_id=ZOCAFE6L3S252W7D6DJG7AO2AJWQMB6VDQLGZ5DE2AU45XMAYBAA, seg_pos={0 0}, upload=788]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:41.280902777 (50m57.061836155s) [upload=788]: Next returned nil segment Jul 26 22:42:41.280933957 (50m57.061867335s) [upload=788]: Waiting for error group managing segments... Jul 26 22:42:41.297380287 (50m57.078313659s) [noise=true , address=104.36.82.52:4091 , piece_id=OPAS4F56AZ6MTZG7IGHHHPT3EHFSZKMV6R4BMS4WI2YGBDWAFTHA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.310454552 (50m57.091396903s) [noise=true , address=77.90.31.21:4900 , piece_id=BPIJVCVJFO2VULVEZCVXKZZJYK4T3ZAQR5SDR2YUED3WVGIHNRBA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.316342461 (50m57.097275827s) [noise=true , address=77.90.31.11:4560 , piece_id=2G27HVSVC2D7VMTF7UEKQ46CQOVXOWFYKV6G7R6RQNRJUKHY5O6Q, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.519159489 (50m57.300092855s) [noise=true , address=79.127.219.38:4004 , piece_id=5LNGIYND5433E5Y6RIXUN5SJ2KZBHAL4VKNM2BETGAJI4ZDRNZJA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.803436354 (50m57.584369716s) [noise=true , address=79.127.205.233:4649 , piece_id=7L55UVAVWQFODXIYC6QBFLLVDSD6RZBXVAGJ2KGKTBVMPLYRIXCQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.829993125 (50m57.610926504s) [noise=true , address=79.127.205.234:4603 , piece_id=AHNYCS34ZHS7OYZGUNE4GRA2UYBZOTSG6BWN37RROM2AFHKTGEXQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.844040237 (50m57.624973606s) [noise=true , address=79.127.205.245:4243 , piece_id=DIWMJGHBI4X3UH6VZNFOBET3FC3GN5JZXMTEQKIXDDNG5HPA7MQA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:41.854421567 (50m57.635355015s) [noise=true , address=79.127.219.48:4090 , piece_id=N4YY2MEHQ5GDG5IEUHRZUYCFZEFKWWB6Z746X2BZ3DT53NDECIHA, seg_pos={0 0}, upload=788]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:42.923305857 (50m58.704239235s) [noise=true , address=79.127.220.101:4048 , piece_id=FXAMGR6YV5ROTM4XXV4HIXLSLB6IUJDF5NBABCLGES6IWI6RNJXQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:43.143184591 (50m58.924117958s) [noise=true , address=45.91.116.11:4124 , piece_id=KK2HXQVYAS2WPWBPO3RDXSJH5AJMJIHI45EKAJ4JXMSMYGUOOXWA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:44.504316917 (51m0.285250347s) [noise=true , address=156.146.43.225:4448 , piece_id=NICLKW2I7FRRO74H2RTUDJNR2XCONV74OAJDEZX52WQPS4PG2VFQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:52.128453795 (51m7.909387163s) [noise=true , address=45.91.116.5:4310 , piece_id=24PKHLS7LIWT5FSSOK6GTAGJA2QT23FGZBTYKCFRRQZIAFUOLJRQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:53.288301846 (51m9.069235229s) [noise=true , address=77.90.31.16:4743 , piece_id=FJ6A2JBU2QRMBBR3S3X6PODEXDDVQ4K534JZYDPAHK2TBVLOO2GA, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:42:57.609070108 (51m13.390003477s) [noise=true , address=45.91.116.10:4100 , piece_id=R2OMVYXOKMSAKXYLG36Z5VBRZQJ47ZXS5EXWOMBUT37JJVX4LMEQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:43:06.108143557 (51m21.889076935s) [noise=true , address=79.127.201.215:4347 , piece_id=2YL6SRDVNRLUVTY52NPSD45CSI6GWLRVGFWA3CJO5AOU2INO7EBQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: <nil> Jul 26 22:43:41.085338392 (51m56.866271767s) [noise=true , address=77.90.31.15:4715 , piece_id=VTL7LRR2FDDRNJMNG5SZDCTLL6NAUAEII52JPV2KSXRDANZ3LWIQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:43:41.085668838 (51m56.866602214s) [noise=true , address=104.36.82.44:4877 , piece_id=45N5BVVXQNCYEQTWEY5UE26PMYX34EZTXAU4IZZTUTMUNR7FFYRQ, seg_pos={0 0}, upload=788]: Done uploading piece. err: ecclient: upload failed (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH, address:104.36.82.44:4877): protocol: rpc: dial tcp 104.36.82.44:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:41.085695001 (51m56.866628379s) [seg_pos={0 0}, upload=788]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:43:41.110295795 (51m56.891229164s) [upload=788]: Done uploading segments... Jul 26 22:43:41.110322252 (51m56.891255627s) [upload=788]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.026947669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 204.802569ms FAIL (2025-07-26 22:43:36 UTC) orbiter warning: uplink: too many requests Duration: 204.831684ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.935422751s FAIL (2025-07-26 22:43:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.935445078s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXX..X..XX..X..XX.X....X....X........X......................] duration: 420.643964ms FAIL (2025-07-26 22:43:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 420.672891ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XXX.XXX.X.XX..X..X......X.............X................X..] duration: 1.123526498s FAIL (2025-07-26 22:43:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.123559369s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........................X........X.X.X...XX.................] duration: 9.923288844s FAIL (2025-07-26 22:43:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.923314902s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 474.242069ms FAIL (2025-07-26 22:43:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 474.269197ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 191.482832ms FAIL (2025-07-26 22:42:47 UTC) orbiter warning: uplink: too many requests Duration: 191.517994ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 45 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.X] duration: 1m0.025796091s FAIL (2025-07-26 22:42:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== m_ObjectCommit Jul 26 22:41:16.892795965 (49m32.673729353s) [upload=766]: Done uploading segments... Jul 26 22:41:16.892822285 (49m32.673755654s) [upload=766]: Upload finished. err: <nil> Jul 26 22:41:31.904533772 (49m47.685467151s) [upload=770]: Starting upload Jul 26 22:41:31.904610378 (49m47.685543748s) [upload=770]: Done starting upload Jul 26 22:41:31.904903375 (49m47.685836742s) [upload=770]: Uploading segments... Jul 26 22:41:31.904947752 (49m47.685881138s) [upload=770]: Got next segment. Inline: false Jul 26 22:41:31.904987837 (49m47.685921213s) [upload=770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:41:31.904993788 (49m47.685927164s) [upload=770]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:41:31.939979572 (49m47.720912958s) [seg_pos={0 0}, upload=770]: Begin upload segment... Jul 26 22:41:31.940234217 (49m47.721167606s) [seg_pos={0 0}, upload=770]: Done begin upload segment. Jul 26 22:41:31.940340475 (49m47.721273864s) [noise=true , address=64.49.13.155:4918 , piece_id=CJC2ZI433LVDE62IBMYGVMUUITLHAFAODXKU6HCUYXKFOS2VRWYA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.940369356 (49m47.721302744s) [noise=true , address=79.127.205.233:4652 , piece_id=SGTPOM6MBVONXYTINHWRXQZDICFXP63DBIN3RC6C7PX7YEV2UWAA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.940917768 (49m47.721851156s) [noise=true , address=64.49.13.140:4508 , piece_id=H67IHSWU6PIPBYYXFPW6JA2XA5WFR6YTJAULCCDYOIFJD7CQPMUQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.941014729 (49m47.721948115s) [noise=true , address=79.127.219.38:4015 , piece_id=ZBU5IXOKKTUA32R3SKRSUZLIN72A3BQM5CSZR3UKMFEG24CSPX3Q, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.941397015 (49m47.722330402s) [noise=true , address=79.127.201.215:4349 , piece_id=LHH6RGB6Z2IS2GAJLVMSJY2VINPH525LJ2XQVFYU6U4B2FMRMCZQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.941616501 (49m47.722549889s) [noise=true , address=79.127.219.47:4061 , piece_id=O5Z43F7YCUWOC3M6N5Q4AOBHOCVYDU4TXYGFBKF3CA4EQS6O4XHQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.941841662 (49m47.722775049s) [noise=true , address=45.91.116.9:4269 , piece_id=6W4AHYW2RTYCJFLUWWEQVMXIAAXMG4UPQZFLHIKH7BUJOLVGFZWQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.942214512 (49m47.723147899s) [noise=true , address=64.49.13.138:4483 , piece_id=TLODN3U5S237CIEQ2DTAFCCVATLQPSOZIT6I7LE3QTQON44FSMGQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.942250992 (49m47.723184378s) [noise=true , address=77.90.31.12:4601 , piece_id=BVGAHU3DT4N6WSQDH6OMZMPGZZAN7FUZJIUSQV5OAWL7UKHPPVVA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.942698123 (49m47.723631511s) [noise=true , address=77.90.31.10:4545 , piece_id=U2RLVAI42C4H7DWORUHEHTQHDMRC4VBTIVPGQM32GSRC32YPWNOQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.942724421 (49m47.723657807s) [noise=true , address=77.90.31.11:4571 , piece_id=YGROPLWOPWHEP63LQNHLRFXBTCT242IEBD7QS3INRZDLAGIHOSYA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.943115112 (49m47.724048499s) [noise=true , address=77.90.31.13:4659 , piece_id=3FLC6YAKNVCOEIKINJONB6ZOEFWKSQDY43WGOROWP3LW5CTQVWJQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.943267669 (49m47.724201054s) [noise=true , address=156.146.43.225:4448 , piece_id=ROPLKCB3V7NOI6I7PAUV6V3CV44LYY2XM2S4N4GMX2NW55HIPGXA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.943676721 (49m47.724610108s) [noise=true , address=64.49.13.170:4719 , piece_id=FF5VCNAKKEZDNK52HLYV3AN72SX4DA3WWUIWRDUM7ZZXC7VJP7JQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.944119065 (49m47.725052454s) [noise=true , address=79.127.205.234:4591 , piece_id=BIVT2MMYNKGMZ3AUUZARNE3PTYBUDFPY4FCXCESHZHM6LZHYFBEQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.944536652 (49m47.72547004s) [noise=true , address=79.127.205.246:4273 , piece_id=ETVT4DT33BC5FKBW4UBSLVYUHIKPEJZ2O2IAAQWNXAWRYIBFUSSQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.946399497 (49m47.727332882s) [noise=true , address=79.127.219.48:4090 , piece_id=EYKSSYL5KBMAEX56CIS6QSB4MLJLBWYYIA25P45BCGCR47VFGASQ, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.946962143 (49m47.72789553s) [noise=true , address=45.91.116.5:4310 , piece_id=7KIIN5J4UGRI4EZPNLQC262CCGXBZCR6GWKVIH2CL5EXHDUMJJOA, seg_pos={0 0}, upload=770]: Uploading piece... Jul 26 22:41:31.992845394 (49m47.77377876s) [noise=true , address=79.127.219.47:4061 , piece_id=O5Z43F7YCUWOC3M6N5Q4AOBHOCVYDU4TXYGFBKF3CA4EQS6O4XHQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:32.065203529 (49m47.846136918s) [upload=770]: Next returned nil segment Jul 26 22:41:32.065227477 (49m47.846160866s) [upload=770]: Waiting for error group managing segments... Jul 26 22:41:32.071542785 (49m47.852476163s) [noise=true , address=79.127.219.38:4015 , piece_id=ZBU5IXOKKTUA32R3SKRSUZLIN72A3BQM5CSZR3UKMFEG24CSPX3Q, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:32.089074327 (49m47.870007692s) [noise=true , address=77.90.31.10:4545 , piece_id=U2RLVAI42C4H7DWORUHEHTQHDMRC4VBTIVPGQM32GSRC32YPWNOQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:32.092848526 (49m47.873781893s) [noise=true , address=156.146.43.225:4448 , piece_id=ROPLKCB3V7NOI6I7PAUV6V3CV44LYY2XM2S4N4GMX2NW55HIPGXA, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:32.131668532 (49m47.912601913s) [noise=true , address=77.90.31.11:4571 , piece_id=YGROPLWOPWHEP63LQNHLRFXBTCT242IEBD7QS3INRZDLAGIHOSYA, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:32.279396058 (49m48.060329441s) [noise=true , address=79.127.219.48:4090 , piece_id=EYKSSYL5KBMAEX56CIS6QSB4MLJLBWYYIA25P45BCGCR47VFGASQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:32.305220249 (49m48.086153626s) [noise=true , address=64.49.13.155:4918 , piece_id=CJC2ZI433LVDE62IBMYGVMUUITLHAFAODXKU6HCUYXKFOS2VRWYA, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:1YjPvRvhJkNJtmt5CJ7nUst9NicEGWzpe7Pn1GWes5uc9H1gTK, address:64.49.13.155:4918): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:32.518001703 (49m48.29893507s) [noise=true , address=79.127.205.234:4591 , piece_id=BIVT2MMYNKGMZ3AUUZARNE3PTYBUDFPY4FCXCESHZHM6LZHYFBEQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:32.553910144 (49m48.334843522s) [noise=true , address=79.127.205.233:4652 , piece_id=SGTPOM6MBVONXYTINHWRXQZDICFXP63DBIN3RC6C7PX7YEV2UWAA, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:32.779618896 (49m48.560552265s) [noise=true , address=79.127.205.246:4273 , piece_id=ETVT4DT33BC5FKBW4UBSLVYUHIKPEJZ2O2IAAQWNXAWRYIBFUSSQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:33.359278311 (49m49.140211675s) [noise=true , address=45.91.116.9:4269 , piece_id=6W4AHYW2RTYCJFLUWWEQVMXIAAXMG4UPQZFLHIKH7BUJOLVGFZWQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:34.554009667 (49m50.334943054s) [noise=true , address=45.91.116.5:4310 , piece_id=7KIIN5J4UGRI4EZPNLQC262CCGXBZCR6GWKVIH2CL5EXHDUMJJOA, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:37.333217000 (49m53.114150378s) [noise=true , address=64.49.13.170:4719 , piece_id=FF5VCNAKKEZDNK52HLYV3AN72SX4DA3WWUIWRDUM7ZZXC7VJP7JQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:38.782141186 (49m54.563074564s) [noise=true , address=64.49.13.140:4508 , piece_id=H67IHSWU6PIPBYYXFPW6JA2XA5WFR6YTJAULCCDYOIFJD7CQPMUQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:39.873142320 (49m55.654075689s) [noise=true , address=64.49.13.138:4483 , piece_id=TLODN3U5S237CIEQ2DTAFCCVATLQPSOZIT6I7LE3QTQON44FSMGQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:41:54.209822084 (50m9.990755464s) [noise=true , address=79.127.201.215:4349 , piece_id=LHH6RGB6Z2IS2GAJLVMSJY2VINPH525LJ2XQVFYU6U4B2FMRMCZQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:42:02.361295715 (50m18.142229099s) [noise=true , address=77.90.31.12:4601 , piece_id=BVGAHU3DT4N6WSQDH6OMZMPGZZAN7FUZJIUSQV5OAWL7UKHPPVVA, seg_pos={0 0}, upload=770]: Done uploading piece. err: <nil> Jul 26 22:42:31.875460070 (50m47.656393436s) [noise=true , address=77.90.31.13:4659 , piece_id=3FLC6YAKNVCOEIKINJONB6ZOEFWKSQDY43WGOROWP3LW5CTQVWJQ, seg_pos={0 0}, upload=770]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: write tcp 159.223.143.39:55040->77.90.31.13:4659: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:42:31.875494050 (50m47.656427425s) [seg_pos={0 0}, upload=770]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:42:31.897672595 (50m47.678605975s) [upload=770]: Done uploading segments... Jul 26 22:42:31.897692998 (50m47.678626381s) [upload=770]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.025819598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [...XXX.XXX.X.XX..X..X......X.............X................X.] duration: 405.483793ms FAIL (2025-07-26 22:42:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 405.517656ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........................X........X.X.X...XX................] duration: 13.924944109s FAIL (2025-07-26 22:42:31 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.924971217s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXX..X..XX..X..XX.X....X....X........X.....................] duration: 583.470676ms FAIL (2025-07-26 22:42:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 583.516144ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 673.218567ms FAIL (2025-07-26 22:42:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 673.25958ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........................X........X.X.X...XX...............] duration: 44.361372717s FAIL (2025-07-26 22:42:17 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.361424826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 14 out of 60 is failed (threshold=3) [....XXX.XXX.X.XX..X..X......X.............X................X] duration: 30.033467865s FAIL (2025-07-26 22:41:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.033498193s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XXX..X..XX..X..XX.X....X....X........X....................] duration: 470.064914ms FAIL (2025-07-26 22:41:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.106652ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 235.749699ms FAIL (2025-07-26 22:41:47 UTC) orbiter warning: uplink: too many requests Duration: 235.778412ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...........................X........X.X.X...XX..............] duration: 25.962093869s FAIL (2025-07-26 22:41:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.962126387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 527.782355ms FAIL (2025-07-26 22:41:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 527.81373ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.] duration: 7.577289004s FAIL (2025-07-26 22:41:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 26 22:40:08.910912462 (48m24.691845855s) [upload=733]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Jul 26 22:40:09.915432325 (48m25.696365696s) [upload=751]: Starting upload Jul 26 22:40:09.915584484 (48m25.696517863s) [upload=751]: Done starting upload Jul 26 22:40:09.915753736 (48m25.696687122s) [upload=751]: Uploading segments... Jul 26 22:40:09.916091771 (48m25.697025143s) [upload=751]: Got next segment. Inline: false Jul 26 22:40:09.916255900 (48m25.69718928s) [upload=751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:40:09.916265701 (48m25.697199087s) [upload=751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:40:09.951897011 (48m25.732830395s) [seg_pos={0 0}, upload=751]: Begin upload segment... Jul 26 22:40:09.951971898 (48m25.732905275s) [seg_pos={0 0}, upload=751]: Done begin upload segment. Jul 26 22:40:09.952009426 (48m25.732942831s) [noise=true , address=79.127.219.47:4048 , piece_id=3XLKJ3TFG5VPRH72DHQVY43Z4YCPQTIEMIIZ5OECNPT3GBYU2JBQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952301126 (48m25.73323451s) [noise=true , address=193.149.164.143:4702 , piece_id=3B5Q4ZNGKLNR3KR462B2M5RJ6Q5MV4YMHKE4TEKDP7BK7SLHTV6Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952758802 (48m25.733692182s) [noise=true , address=104.36.82.11:4128 , piece_id=VMYPCTBSSZAGY6LJ37SMGOZ4JG3LF4ZY37FNUMZ7SPIKJYCOD2XA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952918304 (48m25.733851684s) [noise=true , address=79.127.201.215:4346 , piece_id=2J4IPB3KOKYNZND6M7UM2Z6N6KQEDFICFUGG6VITRLOFXGFUX5YQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953453113 (48m25.734386505s) [noise=true , address=104.36.82.11:4126 , piece_id=NRXAD3XG2NTYTF34MS6COECYG5MAYZB3SCGRGO3GUQARKPWMDQYA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953499878 (48m25.734433267s) [noise=true , address=64.49.13.170:4718 , piece_id=H7FHQNRUITABERXO25B4VIFMA7MHI7WITF35M5Z2W5X2J3MMRWBA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953960495 (48m25.734893878s) [noise=true , address=79.127.205.246:4284 , piece_id=MF2AMEP4HEYFDPNR56T334XLNPCBCEDMUX5A6M4NGFLFQATYNDPQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954069989 (48m25.735003423s) [noise=true , address=79.127.205.234:4615 , piece_id=OMMVZY5FYCWXJWAOGGNZLGSBBK6EDXAOF5AKGNQJUR5HTBRQP5AA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954610643 (48m25.73554403s) [noise=true , address=79.127.205.233:4649 , piece_id=AJ3GJHEJVOYRDV2MDNCKRI6RGAWPCYEWFOXUBVZKWX6EATNW6MNQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954666256 (48m25.735599648s) [noise=true , address=156.146.43.225:4449 , piece_id=CAU3I4RIWPVL7A3BUYMOAOTUMUTM7VS5UMJSAXKQD3U3POTTSCAA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955031508 (48m25.735964894s) [noise=true , address=79.127.219.51:4210 , piece_id=PUGY4EABLOOS7JI3G4VWAJG3FWQNVKIQN6F33FBT23KF7XCAQMAQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955294354 (48m25.736227735s) [noise=true , address=79.127.205.245:4252 , piece_id=4PTGQYPDPYRSXVWWBIOYEGGHCURNHIERME4N3G3YIGOM5LB5MVPA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955495328 (48m25.736428714s) [noise=true , address=79.127.205.235:4689 , piece_id=3GYWKGWSJVVIRIYA7MMHIQ2ICPPM6SXBQ5WBT7B5HVTT4UWANN6Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955851690 (48m25.736785076s) [noise=true , address=77.90.31.17:4800 , piece_id=4ZHOMLKPLJ2AQJA7GKFY3CWA53UVIUD6MZVWGGFC7E35DY5SSA7Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955925977 (48m25.73685936s) [noise=true , address=45.91.116.14:4222 , piece_id=QPG5O763WPESAJS7EPW7QKZBOAO3ELEFRPUD6DYTETVISSMQMPAA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.956391693 (48m25.737325078s) [noise=true , address=104.36.82.2:4755 , piece_id=4MESSBGPPVRO6IQIZUFG4J6PRFPPKCH3AXIJVPFPOZZ3XRMNTRWA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.956928609 (48m25.737861989s) [noise=true , address=79.127.219.48:4075 , piece_id=TSPDPWCCWO6L3ZKO26NU3RXVZK6J7WNF2F3MCRRRUK2JQFMSIBDA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.957437578 (48m25.738370965s) [noise=true , address=77.90.31.14:4684 , piece_id=IU4OAXNVIMFE45KML4GUMQDRHEZSKGTJPK5DEWSEOMBWYHJMDIYQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:10.010392040 (48m25.791325428s) [noise=true , address=79.127.219.51:4210 , piece_id=PUGY4EABLOOS7JI3G4VWAJG3FWQNVKIQN6F33FBT23KF7XCAQMAQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:40:10.075537395 (48m25.856470757s) [upload=751]: Next returned nil segment Jul 26 22:40:10.075558705 (48m25.856492082s) [upload=751]: Waiting for error group managing segments... Jul 26 22:40:10.126404804 (48m25.907338204s) [noise=true , address=77.90.31.17:4800 , piece_id=4ZHOMLKPLJ2AQJA7GKFY3CWA53UVIUD6MZVWGGFC7E35DY5SSA7Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.579064023 (48m26.359997387s) [noise=true , address=79.127.205.246:4284 , piece_id=MF2AMEP4HEYFDPNR56T334XLNPCBCEDMUX5A6M4NGFLFQATYNDPQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.592223660 (48m26.373157044s) [noise=true , address=79.127.205.235:4689 , piece_id=3GYWKGWSJVVIRIYA7MMHIQ2ICPPM6SXBQ5WBT7B5HVTT4UWANN6Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.625813526 (48m26.406746892s) [noise=true , address=79.127.205.234:4615 , piece_id=OMMVZY5FYCWXJWAOGGNZLGSBBK6EDXAOF5AKGNQJUR5HTBRQP5AA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.860506119 (48m26.641439485s) [noise=true , address=79.127.205.245:4252 , piece_id=4PTGQYPDPYRSXVWWBIOYEGGHCURNHIERME4N3G3YIGOM5LB5MVPA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.873202746 (48m26.654136111s) [noise=true , address=79.127.205.233:4649 , piece_id=AJ3GJHEJVOYRDV2MDNCKRI6RGAWPCYEWFOXUBVZKWX6EATNW6MNQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:11.115326185 (48m26.896259588s) [noise=true , address=193.149.164.143:4702 , piece_id=3B5Q4ZNGKLNR3KR462B2M5RJ6Q5MV4YMHKE4TEKDP7BK7SLHTV6Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Jul 26 22:40:11.717107067 (48m27.498040445s) [noise=true , address=79.127.219.48:4075 , piece_id=TSPDPWCCWO6L3ZKO26NU3RXVZK6J7WNF2F3MCRRRUK2JQFMSIBDA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:12.148650635 (48m27.929584023s) [noise=true , address=79.127.219.47:4048 , piece_id=3XLKJ3TFG5VPRH72DHQVY43Z4YCPQTIEMIIZ5OECNPT3GBYU2JBQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:13.464654731 (48m29.24558811s) [noise=true , address=64.49.13.170:4718 , piece_id=H7FHQNRUITABERXO25B4VIFMA7MHI7WITF35M5Z2W5X2J3MMRWBA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:14.660571356 (48m30.441504723s) [noise=true , address=156.146.43.225:4449 , piece_id=CAU3I4RIWPVL7A3BUYMOAOTUMUTM7VS5UMJSAXKQD3U3POTTSCAA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:24.682119347 (48m40.463052728s) [noise=true , address=77.90.31.14:4684 , piece_id=IU4OAXNVIMFE45KML4GUMQDRHEZSKGTJPK5DEWSEOMBWYHJMDIYQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:25.404814950 (48m41.185748318s) [noise=true , address=45.91.116.14:4222 , piece_id=QPG5O763WPESAJS7EPW7QKZBOAO3ELEFRPUD6DYTETVISSMQMPAA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:59.474246858 (49m15.255180221s) [noise=true , address=79.127.201.215:4346 , piece_id=2J4IPB3KOKYNZND6M7UM2Z6N6KQEDFICFUGG6VITRLOFXGFUX5YQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:41:09.882265336 (49m25.663198715s) [noise=true , address=104.36.82.11:4128 , piece_id=VMYPCTBSSZAGY6LJ37SMGOZ4JG3LF4ZY37FNUMZ7SPIKJYCOD2XA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS, address:104.36.82.11:4128): protocol: write tcp 159.223.143.39:60630->104.36.82.11:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882373704 (49m25.663307086s) [noise=true , address=104.36.82.11:4126 , piece_id=NRXAD3XG2NTYTF34MS6COECYG5MAYZB3SCGRGO3GUQARKPWMDQYA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:12R3SVXrhJsmYTXR4vDSkfaiqDRox21u17SLmQc9sEab1B7zZRr, address:104.36.82.11:4126): protocol: write tcp 159.223.143.39:51650->104.36.82.11:4126: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882429118 (49m25.663362498s) [noise=true , address=104.36.82.2:4755 , piece_id=4MESSBGPPVRO6IQIZUFG4J6PRFPPKCH3AXIJVPFPOZZ3XRMNTRWA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: write tcp 159.223.143.39:34790->104.36.82.2:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882450347 (49m25.663383729s) [seg_pos={0 0}, upload=751]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:41:09.906834186 (49m25.687767566s) [upload=751]: Done uploading segments... Jul 26 22:41:09.906864582 (49m25.687797962s) [upload=751]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.577315388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..XXX..X..XX..X..XX.X....X....X........X...................] duration: 467.803741ms FAIL (2025-07-26 22:41:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 467.841174ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX] duration: 1m0.038348053s FAIL (2025-07-26 22:41:09 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 26 22:40:08.910912462 (48m24.691845855s) [upload=733]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Jul 26 22:40:09.915432325 (48m25.696365696s) [upload=751]: Starting upload Jul 26 22:40:09.915584484 (48m25.696517863s) [upload=751]: Done starting upload Jul 26 22:40:09.915753736 (48m25.696687122s) [upload=751]: Uploading segments... Jul 26 22:40:09.916091771 (48m25.697025143s) [upload=751]: Got next segment. Inline: false Jul 26 22:40:09.916255900 (48m25.69718928s) [upload=751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:40:09.916265701 (48m25.697199087s) [upload=751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:40:09.951897011 (48m25.732830395s) [seg_pos={0 0}, upload=751]: Begin upload segment... Jul 26 22:40:09.951971898 (48m25.732905275s) [seg_pos={0 0}, upload=751]: Done begin upload segment. Jul 26 22:40:09.952009426 (48m25.732942831s) [noise=true , address=79.127.219.47:4048 , piece_id=3XLKJ3TFG5VPRH72DHQVY43Z4YCPQTIEMIIZ5OECNPT3GBYU2JBQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952301126 (48m25.73323451s) [noise=true , address=193.149.164.143:4702 , piece_id=3B5Q4ZNGKLNR3KR462B2M5RJ6Q5MV4YMHKE4TEKDP7BK7SLHTV6Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952758802 (48m25.733692182s) [noise=true , address=104.36.82.11:4128 , piece_id=VMYPCTBSSZAGY6LJ37SMGOZ4JG3LF4ZY37FNUMZ7SPIKJYCOD2XA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.952918304 (48m25.733851684s) [noise=true , address=79.127.201.215:4346 , piece_id=2J4IPB3KOKYNZND6M7UM2Z6N6KQEDFICFUGG6VITRLOFXGFUX5YQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953453113 (48m25.734386505s) [noise=true , address=104.36.82.11:4126 , piece_id=NRXAD3XG2NTYTF34MS6COECYG5MAYZB3SCGRGO3GUQARKPWMDQYA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953499878 (48m25.734433267s) [noise=true , address=64.49.13.170:4718 , piece_id=H7FHQNRUITABERXO25B4VIFMA7MHI7WITF35M5Z2W5X2J3MMRWBA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.953960495 (48m25.734893878s) [noise=true , address=79.127.205.246:4284 , piece_id=MF2AMEP4HEYFDPNR56T334XLNPCBCEDMUX5A6M4NGFLFQATYNDPQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954069989 (48m25.735003423s) [noise=true , address=79.127.205.234:4615 , piece_id=OMMVZY5FYCWXJWAOGGNZLGSBBK6EDXAOF5AKGNQJUR5HTBRQP5AA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954610643 (48m25.73554403s) [noise=true , address=79.127.205.233:4649 , piece_id=AJ3GJHEJVOYRDV2MDNCKRI6RGAWPCYEWFOXUBVZKWX6EATNW6MNQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.954666256 (48m25.735599648s) [noise=true , address=156.146.43.225:4449 , piece_id=CAU3I4RIWPVL7A3BUYMOAOTUMUTM7VS5UMJSAXKQD3U3POTTSCAA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955031508 (48m25.735964894s) [noise=true , address=79.127.219.51:4210 , piece_id=PUGY4EABLOOS7JI3G4VWAJG3FWQNVKIQN6F33FBT23KF7XCAQMAQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955294354 (48m25.736227735s) [noise=true , address=79.127.205.245:4252 , piece_id=4PTGQYPDPYRSXVWWBIOYEGGHCURNHIERME4N3G3YIGOM5LB5MVPA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955495328 (48m25.736428714s) [noise=true , address=79.127.205.235:4689 , piece_id=3GYWKGWSJVVIRIYA7MMHIQ2ICPPM6SXBQ5WBT7B5HVTT4UWANN6Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955851690 (48m25.736785076s) [noise=true , address=77.90.31.17:4800 , piece_id=4ZHOMLKPLJ2AQJA7GKFY3CWA53UVIUD6MZVWGGFC7E35DY5SSA7Q, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.955925977 (48m25.73685936s) [noise=true , address=45.91.116.14:4222 , piece_id=QPG5O763WPESAJS7EPW7QKZBOAO3ELEFRPUD6DYTETVISSMQMPAA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.956391693 (48m25.737325078s) [noise=true , address=104.36.82.2:4755 , piece_id=4MESSBGPPVRO6IQIZUFG4J6PRFPPKCH3AXIJVPFPOZZ3XRMNTRWA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.956928609 (48m25.737861989s) [noise=true , address=79.127.219.48:4075 , piece_id=TSPDPWCCWO6L3ZKO26NU3RXVZK6J7WNF2F3MCRRRUK2JQFMSIBDA, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:09.957437578 (48m25.738370965s) [noise=true , address=77.90.31.14:4684 , piece_id=IU4OAXNVIMFE45KML4GUMQDRHEZSKGTJPK5DEWSEOMBWYHJMDIYQ, seg_pos={0 0}, upload=751]: Uploading piece... Jul 26 22:40:10.010392040 (48m25.791325428s) [noise=true , address=79.127.219.51:4210 , piece_id=PUGY4EABLOOS7JI3G4VWAJG3FWQNVKIQN6F33FBT23KF7XCAQMAQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:40:10.075537395 (48m25.856470757s) [upload=751]: Next returned nil segment Jul 26 22:40:10.075558705 (48m25.856492082s) [upload=751]: Waiting for error group managing segments... Jul 26 22:40:10.126404804 (48m25.907338204s) [noise=true , address=77.90.31.17:4800 , piece_id=4ZHOMLKPLJ2AQJA7GKFY3CWA53UVIUD6MZVWGGFC7E35DY5SSA7Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.579064023 (48m26.359997387s) [noise=true , address=79.127.205.246:4284 , piece_id=MF2AMEP4HEYFDPNR56T334XLNPCBCEDMUX5A6M4NGFLFQATYNDPQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.592223660 (48m26.373157044s) [noise=true , address=79.127.205.235:4689 , piece_id=3GYWKGWSJVVIRIYA7MMHIQ2ICPPM6SXBQ5WBT7B5HVTT4UWANN6Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.625813526 (48m26.406746892s) [noise=true , address=79.127.205.234:4615 , piece_id=OMMVZY5FYCWXJWAOGGNZLGSBBK6EDXAOF5AKGNQJUR5HTBRQP5AA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.860506119 (48m26.641439485s) [noise=true , address=79.127.205.245:4252 , piece_id=4PTGQYPDPYRSXVWWBIOYEGGHCURNHIERME4N3G3YIGOM5LB5MVPA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:10.873202746 (48m26.654136111s) [noise=true , address=79.127.205.233:4649 , piece_id=AJ3GJHEJVOYRDV2MDNCKRI6RGAWPCYEWFOXUBVZKWX6EATNW6MNQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:11.115326185 (48m26.896259588s) [noise=true , address=193.149.164.143:4702 , piece_id=3B5Q4ZNGKLNR3KR462B2M5RJ6Q5MV4YMHKE4TEKDP7BK7SLHTV6Q, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Jul 26 22:40:11.717107067 (48m27.498040445s) [noise=true , address=79.127.219.48:4075 , piece_id=TSPDPWCCWO6L3ZKO26NU3RXVZK6J7WNF2F3MCRRRUK2JQFMSIBDA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:12.148650635 (48m27.929584023s) [noise=true , address=79.127.219.47:4048 , piece_id=3XLKJ3TFG5VPRH72DHQVY43Z4YCPQTIEMIIZ5OECNPT3GBYU2JBQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:13.464654731 (48m29.24558811s) [noise=true , address=64.49.13.170:4718 , piece_id=H7FHQNRUITABERXO25B4VIFMA7MHI7WITF35M5Z2W5X2J3MMRWBA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:14.660571356 (48m30.441504723s) [noise=true , address=156.146.43.225:4449 , piece_id=CAU3I4RIWPVL7A3BUYMOAOTUMUTM7VS5UMJSAXKQD3U3POTTSCAA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:24.682119347 (48m40.463052728s) [noise=true , address=77.90.31.14:4684 , piece_id=IU4OAXNVIMFE45KML4GUMQDRHEZSKGTJPK5DEWSEOMBWYHJMDIYQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:25.404814950 (48m41.185748318s) [noise=true , address=45.91.116.14:4222 , piece_id=QPG5O763WPESAJS7EPW7QKZBOAO3ELEFRPUD6DYTETVISSMQMPAA, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:40:59.474246858 (49m15.255180221s) [noise=true , address=79.127.201.215:4346 , piece_id=2J4IPB3KOKYNZND6M7UM2Z6N6KQEDFICFUGG6VITRLOFXGFUX5YQ, seg_pos={0 0}, upload=751]: Done uploading piece. err: <nil> Jul 26 22:41:09.882265336 (49m25.663198715s) [noise=true , address=104.36.82.11:4128 , piece_id=VMYPCTBSSZAGY6LJ37SMGOZ4JG3LF4ZY37FNUMZ7SPIKJYCOD2XA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1wABkAtAjoYkKyNoAiaVCxSvY6WgvHvAYFPCjpNojtpiYiGtDS, address:104.36.82.11:4128): protocol: write tcp 159.223.143.39:60630->104.36.82.11:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882373704 (49m25.663307086s) [noise=true , address=104.36.82.11:4126 , piece_id=NRXAD3XG2NTYTF34MS6COECYG5MAYZB3SCGRGO3GUQARKPWMDQYA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:12R3SVXrhJsmYTXR4vDSkfaiqDRox21u17SLmQc9sEab1B7zZRr, address:104.36.82.11:4126): protocol: write tcp 159.223.143.39:51650->104.36.82.11:4126: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882429118 (49m25.663362498s) [noise=true , address=104.36.82.2:4755 , piece_id=4MESSBGPPVRO6IQIZUFG4J6PRFPPKCH3AXIJVPFPOZZ3XRMNTRWA, seg_pos={0 0}, upload=751]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: write tcp 159.223.143.39:34790->104.36.82.2:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:41:09.882450347 (49m25.663383729s) [seg_pos={0 0}, upload=751]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:41:09.906834186 (49m25.687767566s) [upload=751]: Done uploading segments... Jul 26 22:41:09.906864582 (49m25.687797962s) [upload=751]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.038381337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX..XXX..X..XX..X..XX.X....X....X........X..................] duration: 542.836713ms FAIL (2025-07-26 22:40:46 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 542.87346ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 346.222654ms FAIL (2025-07-26 22:40:34 UTC) orbiter warning: uplink: too many requests Duration: 346.241481ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.920141279s FAIL (2025-07-26 22:40:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.920193145s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....XXX.XXX.X.XX..X..X......X.............X................] duration: 444.928328ms FAIL (2025-07-26 22:40:17 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 444.985143ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 555.803902ms FAIL (2025-07-26 22:40:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 555.82877ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XX] duration: 1m0.036278761s FAIL (2025-07-26 22:40:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== : Starting upload Jul 26 22:39:08.923788388 (47m24.704721774s) [upload=733]: Done starting upload Jul 26 22:39:08.924128497 (47m24.705061877s) [upload=733]: Uploading segments... Jul 26 22:39:08.924435891 (47m24.705369269s) [upload=733]: Got next segment. Inline: false Jul 26 22:39:08.924507771 (47m24.705441151s) [upload=733]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:39:08.924514132 (47m24.705447515s) [upload=733]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:39:08.961235730 (47m24.742169115s) [seg_pos={0 0}, upload=733]: Begin upload segment... Jul 26 22:39:08.961321571 (47m24.742254956s) [seg_pos={0 0}, upload=733]: Done begin upload segment. Jul 26 22:39:08.961351688 (47m24.742285077s) [noise=true , address=79.127.201.215:4360 , piece_id=NGVCW4MLZSTK2TBEO7OLM7X5MZD6OFGBFJFQC2MCWBSNHRNVRQEQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.961370010 (47m24.742303392s) [noise=true , address=45.91.116.12:4229 , piece_id=WJ3LFHU36TPCZAVYLMWT3JFZF62C6LKBNOC4NZXUVQAQ6G2FESZQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.961860586 (47m24.742793968s) [noise=true , address=79.127.220.101:4048 , piece_id=X55VC4JDHS2FBPBCLNFDJ2OHD72WB65ZFHHGET54EUFHLOB77WDA, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.962269973 (47m24.743203356s) [noise=true , address=79.127.219.48:4108 , piece_id=ZD5SB7BTYATPL6GDJVMPXSLMQG3A2DUWIZPC7WIZSLRL6IOR2GGQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.962721691 (47m24.743655077s) [noise=true , address=77.90.31.12:4612 , piece_id=5BJ7IT4LNU7K2SSTR4VIBBY6KOAHQCXKYW44HA7UPC2KJTMHEY3A, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.963156595 (47m24.744089979s) [noise=true , address=77.90.31.16:4743 , piece_id=J7R3UT7IX5TE4NZIXVNC7UUO6RGELOYKTRG3DHZCDRPOAPPSL62A, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.963549149 (47m24.744482533s) [noise=true , address=156.146.43.225:4448 , piece_id=J7L3DC3T5LIAX4YXO4ZXMNNO5XUCDDMJKJEG5H42VJFOS7SYAQ4A, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.963954262 (47m24.744887645s) [noise=true , address=79.127.219.47:4069 , piece_id=F62NQJ5FL2UID4ZANGIYXVGRCMGIAB3DKMFVVRUQNSFWAW3CXRBQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.968141076 (47m24.749074452s) [noise=true , address=45.91.116.9:4268 , piece_id=I2LSWLBB5KBEX3T7G2VG7JY46LWGHFDAQALKMEYKCBPQ2QYTY2HQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.968650830 (47m24.74958422s) [noise=true , address=64.49.13.170:4721 , piece_id=CN5C77YJYS6EXXRZA2KLJ2DJBBU24IJMIIUBEVDXB224T5ILPMYA, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.969137867 (47m24.750071254s) [noise=true , address=45.91.116.10:4100 , piece_id=CVKC5XCMKVIEFG7MAWWU476NUSS46K7BW46DLCDGXCCXWW3DLTVQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.969641119 (47m24.750574506s) [noise=true , address=79.127.205.235:4682 , piece_id=QKAZRYX7JK3LPDKG7MEBZ3COLUFRYWC5O3BDSA332EOE3XALJ4IA, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.970186290 (47m24.751119677s) [noise=true , address=104.36.82.22:4509 , piece_id=QJWYVVJ2UYTMTECGJLDBUPBVSJQQBXKC55B5CAT3JFOCID36AJAQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.970591719 (47m24.751525106s) [noise=true , address=79.127.219.51:4210 , piece_id=XYEMCDISA33LQU33TIOEXC7TLCZFCPYY2VFNJ4GSJDQJD3PJKXZQ, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.970965884 (47m24.751899272s) [noise=true , address=79.127.205.245:4251 , piece_id=AI64TIXMPEG5AXKWS3IYOCW6ETQ5TNO2ZGEWXPQS2YUVRKNKHQ2Q, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.971409364 (47m24.752342751s) [noise=true , address=64.49.13.155:4923 , piece_id=QJTWA6ZXJASTJQYNLYGG3GTL6E2ZKAOYN6ZGZA7UUGNCT3TS2K7Q, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.971812382 (47m24.752745767s) [noise=true , address=79.127.219.38:4003 , piece_id=X2HHRAODDWWJU4BCSCZLR733BIETPPDHX52XXB3ANGYCQ7IJTXRA, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:08.974320352 (47m24.755253738s) [noise=true , address=79.127.205.234:4606 , piece_id=574OK4SH67GGE77YJJC2JWUWIAPAQK7GWZQ4WV63WAH7LGE23R6A, seg_pos={0 0}, upload=733]: Uploading piece... Jul 26 22:39:09.031814795 (47m24.81274818s) [noise=true , address=45.91.116.9:4268 , piece_id=I2LSWLBB5KBEX3T7G2VG7JY46LWGHFDAQALKMEYKCBPQ2QYTY2HQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:1pe6TJYzVFzoAjLZc7AMNz58WdHcPDQckxUwYn6PgJuEVji9Lt, address:45.91.116.9:4268): protocol: not enough available disk space, have: 0, need: 6725632 Jul 26 22:39:09.032522272 (47m24.813455639s) [noise=true , address=79.127.219.48:4108 , piece_id=ZD5SB7BTYATPL6GDJVMPXSLMQG3A2DUWIZPC7WIZSLRL6IOR2GGQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:1JqrKuRY5vhNV26p42EdfHb5KePUEHnRAUQfna2RLSFz3XXfsG, address:79.127.219.48:4108): protocol: not enough available disk space, have: 0, need: 6725632 Jul 26 22:39:09.078623076 (47m24.859556461s) [upload=733]: Next returned nil segment Jul 26 22:39:09.078644987 (47m24.859578386s) [upload=733]: Waiting for error group managing segments... Jul 26 22:39:09.098122933 (47m24.879056296s) [noise=true , address=77.90.31.12:4612 , piece_id=5BJ7IT4LNU7K2SSTR4VIBBY6KOAHQCXKYW44HA7UPC2KJTMHEY3A, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:09.099379503 (47m24.88031287s) [noise=true , address=79.127.219.51:4210 , piece_id=XYEMCDISA33LQU33TIOEXC7TLCZFCPYY2VFNJ4GSJDQJD3PJKXZQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:39:09.121579166 (47m24.902512531s) [noise=true , address=77.90.31.16:4743 , piece_id=J7R3UT7IX5TE4NZIXVNC7UUO6RGELOYKTRG3DHZCDRPOAPPSL62A, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:09.125866269 (47m24.90679965s) [noise=true , address=79.127.219.38:4003 , piece_id=X2HHRAODDWWJU4BCSCZLR733BIETPPDHX52XXB3ANGYCQ7IJTXRA, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:09.599049558 (47m25.379982931s) [noise=true , address=79.127.205.245:4251 , piece_id=AI64TIXMPEG5AXKWS3IYOCW6ETQ5TNO2ZGEWXPQS2YUVRKNKHQ2Q, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:09.615181047 (47m25.396114416s) [noise=true , address=79.127.205.235:4682 , piece_id=QKAZRYX7JK3LPDKG7MEBZ3COLUFRYWC5O3BDSA332EOE3XALJ4IA, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:09.678010390 (47m25.458943779s) [noise=true , address=79.127.205.234:4606 , piece_id=574OK4SH67GGE77YJJC2JWUWIAPAQK7GWZQ4WV63WAH7LGE23R6A, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:10.408670184 (47m26.189603547s) [noise=true , address=79.127.220.101:4048 , piece_id=X55VC4JDHS2FBPBCLNFDJ2OHD72WB65ZFHHGET54EUFHLOB77WDA, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:10.500255954 (47m26.281189319s) [noise=true , address=79.127.219.47:4069 , piece_id=F62NQJ5FL2UID4ZANGIYXVGRCMGIAB3DKMFVVRUQNSFWAW3CXRBQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:10.657003674 (47m26.437937044s) [noise=true , address=156.146.43.225:4448 , piece_id=J7L3DC3T5LIAX4YXO4ZXMNNO5XUCDDMJKJEG5H42VJFOS7SYAQ4A, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:13.153669640 (47m28.934603018s) [noise=true , address=64.49.13.170:4721 , piece_id=CN5C77YJYS6EXXRZA2KLJ2DJBBU24IJMIIUBEVDXB224T5ILPMYA, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:39:13.224172652 (47m29.005106021s) [noise=true , address=64.49.13.155:4923 , piece_id=QJTWA6ZXJASTJQYNLYGG3GTL6E2ZKAOYN6ZGZA7UUGNCT3TS2K7Q, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:39:40.649544231 (47m56.430477616s) [noise=true , address=45.91.116.12:4229 , piece_id=WJ3LFHU36TPCZAVYLMWT3JFZF62C6LKBNOC4NZXUVQAQ6G2FESZQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: storage node overloaded, request limit: 2000 Jul 26 22:39:50.837446148 (48m6.618379515s) [noise=true , address=79.127.201.215:4360 , piece_id=NGVCW4MLZSTK2TBEO7OLM7X5MZD6OFGBFJFQC2MCWBSNHRNVRQEQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: <nil> Jul 26 22:40:08.882267994 (48m24.663201374s) [noise=true , address=104.36.82.22:4509 , piece_id=QJWYVVJ2UYTMTECGJLDBUPBVSJQQBXKC55B5CAT3JFOCID36AJAQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:104.36.82.22:4509): protocol: rpc: dial tcp 104.36.82.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:40:08.882762173 (48m24.663695535s) [noise=true , address=45.91.116.10:4100 , piece_id=CVKC5XCMKVIEFG7MAWWU476NUSS46K7BW46DLCDGXCCXWW3DLTVQ, seg_pos={0 0}, upload=733]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: write tcp 159.223.143.39:36046->45.91.116.10:4100: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:40:08.882820611 (48m24.663754064s) [seg_pos={0 0}, upload=733]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:40:08.910675515 (48m24.69160888s) [upload=733]: Done uploading segments... Jul 26 22:40:08.910912462 (48m24.691845855s) [upload=733]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.036302273s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [............................X........X.X.X...XX.............] duration: 3.338275413s FAIL (2025-07-26 22:40:08 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.338306565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 204.774258ms FAIL (2025-07-26 22:39:58 UTC) orbiter warning: uplink: too many requests Duration: 204.803531ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX..XXX..X..XX..X..XX.X....X....X........X.................] duration: 415.12892ms FAIL (2025-07-26 22:39:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.159218ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 448.162479ms FAIL (2025-07-26 22:39:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 448.185926ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.............................X........X.X.X...XX............] duration: 10.377429309s FAIL (2025-07-26 22:39:38 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.377452609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 239.278635ms FAIL (2025-07-26 22:39:35 UTC) orbiter warning: uplink: too many requests Duration: 239.333907ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [......XXX.XXX.X.XX..X..X......X.............X...............] duration: 606.118021ms FAIL (2025-07-26 22:39:19 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 606.152692ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX..XXX..X..XX..X..XX.X....X....X........X................] duration: 471.616703ms FAIL (2025-07-26 22:39:18 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 471.654985ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 42 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.X] duration: 1m0.027899098s FAIL (2025-07-26 22:39:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 9636s) [upload=716]: Starting upload Jul 26 22:38:08.835390171 (46m24.616323563s) [upload=716]: Done starting upload Jul 26 22:38:08.835566870 (46m24.616500241s) [upload=716]: Uploading segments... Jul 26 22:38:08.835782913 (46m24.616716297s) [upload=716]: Got next segment. Inline: false Jul 26 22:38:08.835854662 (46m24.61678805s) [upload=716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:38:08.835862516 (46m24.616795901s) [upload=716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:38:08.873069108 (46m24.654002498s) [seg_pos={0 0}, upload=716]: Begin upload segment... Jul 26 22:38:08.873160866 (46m24.654094247s) [seg_pos={0 0}, upload=716]: Done begin upload segment. Jul 26 22:38:08.873209596 (46m24.654142977s) [noise=true , address=79.127.205.245:4248 , piece_id=N4BVIG4RJ2DM4IG2UY5Q5X4ZVQXHZ3MWCMEJ66GSE3ACZXB25HPQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.873808981 (46m24.654742364s) [noise=true , address=77.90.31.13:4634 , piece_id=NLH3ETOODFJ2H4AW4L2VKHH3Y7VCNIXWGJPAO37ACZPGZTGUM7WQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.874263438 (46m24.655196826s) [noise=true , address=156.146.43.225:4448 , piece_id=45LL457ULSE3WSAEENTOMCTUSZXP3G7XFBADGRXT6SEPARNBQK5Q, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.874606330 (46m24.655539697s) [noise=true , address=77.90.31.11:4585 , piece_id=BJBA64B4SMOPD4UH7NAVHOT3IA4ERXRREMIKZBLVGDJ5SZYUOVVQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.875045445 (46m24.655978835s) [noise=true , address=79.127.219.48:4090 , piece_id=DV7WFRTPG7EG55JGYTK4VA4YOKQRU3RBNBJFULZETF7NKVYDD2IQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.875494730 (46m24.656428097s) [noise=true , address=79.127.205.235:4688 , piece_id=5YAGJOEICJQCDDKS4GU6OCBJUTM5SN2HUA4DL6ADIEQEBHDWO4XA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.875823526 (46m24.656756914s) [noise=true , address=45.91.116.5:4310 , piece_id=QXWHPJJT6WFMORQXG3CDZ7TFNAFBFQDAXUKHU6W5WEAYYRWCRP4A, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.876383365 (46m24.657316753s) [noise=true , address=77.90.31.17:4779 , piece_id=TZZVMWKUK4VOOCYXHEKBVGDPB2TCSVTYDB4VD4PTFVWOB56MH4WA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.876740836 (46m24.657674222s) [noise=true , address=79.127.220.101:4048 , piece_id=6OV4FPE6SFDT6WU6JF3TAY5U4GMOBIK4MCJRMCPY7VA7CUZJ5B2Q, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.877379894 (46m24.658313282s) [noise=true , address=77.90.31.18:4839 , piece_id=CWJ4MSY6YK2AZ6XITE2WRON2T54XCYXIDFG5YEKOSBWRAZKUXV7Q, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.877749557 (46m24.658682947s) [noise=true , address=79.127.201.215:4360 , piece_id=5GZZQWBCWIMR455MQ257ETOZ2WEQ6BLYWLA6OHZIQ673EZS4OPCQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.878278845 (46m24.659212234s) [noise=true , address=79.127.219.38:4001 , piece_id=FH3V4IXPZCAQJ43AXVLL3XIEPJSQDPPPOI2ZWV72LL53U5UFFJNA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.878748085 (46m24.659681471s) [noise=true , address=77.90.31.15:4731 , piece_id=A6C73RYM67EGBOCLTAA764AZII636W4PKAZQ6DACRHJYK5RD5GUA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.879280496 (46m24.660213887s) [noise=true , address=79.127.205.246:4266 , piece_id=IM5NV6TLAMIUAWFLW6TVFWU45GRRMWNJAJMWZDQC6V7AT4NQOTLQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.879777351 (46m24.660710737s) [noise=true , address=77.90.31.21:4885 , piece_id=6RU2Y3S7UDHAJDHCHMFEYCMYZHOPU6JSVLXGYQE4WFYJXBP5HUEA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.880280029 (46m24.661213419s) [noise=true , address=193.149.164.143:4702 , piece_id=4HLD6BQVASDLVOKKHDS7GERYXLT43U2UXGJKISW73WF5ZRMIAQSQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.880664135 (46m24.661597516s) [noise=true , address=104.36.82.42:4826 , piece_id=ICX4E7GB2PYOJBILF3754FAL45TBSECQ6RTAH4PVDKM3THPKLZUA, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.881101348 (46m24.662034742s) [noise=true , address=77.90.31.12:4608 , piece_id=5UZVWLWNFJGWBK7AAS4RNMLNU765GXNWFXDMCKFZHJDNEQKGGQOQ, seg_pos={0 0}, upload=716]: Uploading piece... Jul 26 22:38:08.945714663 (46m24.726648049s) [noise=true , address=79.127.219.48:4090 , piece_id=DV7WFRTPG7EG55JGYTK4VA4YOKQRU3RBNBJFULZETF7NKVYDD2IQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:08.945764441 (46m24.726697829s) [noise=true , address=77.90.31.13:4634 , piece_id=NLH3ETOODFJ2H4AW4L2VKHH3Y7VCNIXWGJPAO37ACZPGZTGUM7WQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:1WwFU9dJW2LLdTQ64FLEpt4qyQLVuiEb8PmSLrUokav2HGVCw7, address:77.90.31.13:4634): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:09.004020033 (46m24.784953481s) [upload=716]: Next returned nil segment Jul 26 22:38:09.004042038 (46m24.784975425s) [upload=716]: Waiting for error group managing segments... Jul 26 22:38:09.022616432 (46m24.803549814s) [noise=true , address=156.146.43.225:4448 , piece_id=45LL457ULSE3WSAEENTOMCTUSZXP3G7XFBADGRXT6SEPARNBQK5Q, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:09.028632122 (46m24.809565484s) [noise=true , address=79.127.219.38:4001 , piece_id=FH3V4IXPZCAQJ43AXVLL3XIEPJSQDPPPOI2ZWV72LL53U5UFFJNA, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.043520465 (46m24.824453852s) [noise=true , address=79.127.220.101:4048 , piece_id=6OV4FPE6SFDT6WU6JF3TAY5U4GMOBIK4MCJRMCPY7VA7CUZJ5B2Q, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:38:09.052156291 (46m24.833089666s) [noise=true , address=77.90.31.21:4885 , piece_id=6RU2Y3S7UDHAJDHCHMFEYCMYZHOPU6JSVLXGYQE4WFYJXBP5HUEA, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.061288317 (46m24.842221681s) [noise=true , address=77.90.31.11:4585 , piece_id=BJBA64B4SMOPD4UH7NAVHOT3IA4ERXRREMIKZBLVGDJ5SZYUOVVQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.063141664 (46m24.844075051s) [noise=true , address=77.90.31.12:4608 , piece_id=5UZVWLWNFJGWBK7AAS4RNMLNU765GXNWFXDMCKFZHJDNEQKGGQOQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.093133770 (46m24.874067151s) [noise=true , address=77.90.31.17:4779 , piece_id=TZZVMWKUK4VOOCYXHEKBVGDPB2TCSVTYDB4VD4PTFVWOB56MH4WA, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.465880506 (46m25.246813874s) [noise=true , address=79.127.205.246:4266 , piece_id=IM5NV6TLAMIUAWFLW6TVFWU45GRRMWNJAJMWZDQC6V7AT4NQOTLQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.516371284 (46m25.297304661s) [noise=true , address=79.127.205.235:4688 , piece_id=5YAGJOEICJQCDDKS4GU6OCBJUTM5SN2HUA4DL6ADIEQEBHDWO4XA, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.517320708 (46m25.298254089s) [noise=true , address=79.127.205.245:4248 , piece_id=N4BVIG4RJ2DM4IG2UY5Q5X4ZVQXHZ3MWCMEJ66GSE3ACZXB25HPQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:09.641480962 (46m25.422414347s) [noise=true , address=193.149.164.143:4702 , piece_id=4HLD6BQVASDLVOKKHDS7GERYXLT43U2UXGJKISW73WF5ZRMIAQSQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:38:22.794780350 (46m38.575713727s) [noise=true , address=77.90.31.15:4731 , piece_id=A6C73RYM67EGBOCLTAA764AZII636W4PKAZQ6DACRHJYK5RD5GUA, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:39:05.978548585 (47m21.759481954s) [noise=true , address=79.127.201.215:4360 , piece_id=5GZZQWBCWIMR455MQ257ETOZ2WEQ6BLYWLA6OHZIQ673EZS4OPCQ, seg_pos={0 0}, upload=716]: Done uploading piece. err: <nil> Jul 26 22:39:06.046009209 (47m21.826942594s) [noise=true , address=77.90.31.18:4839 , piece_id=CWJ4MSY6YK2AZ6XITE2WRON2T54XCYXIDFG5YEKOSBWRAZKUXV7Q, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx, address:77.90.31.18:4839): protocol: storage node overloaded, request limit: 2000 Jul 26 22:39:08.795713604 (47m24.576646988s) [noise=true , address=45.91.116.5:4310 , piece_id=QXWHPJJT6WFMORQXG3CDZ7TFNAFBFQDAXUKHU6W5WEAYYRWCRP4A, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: rpc: dial tcp 45.91.116.5:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:39:08.795824905 (47m24.576758299s) [noise=true , address=104.36.82.42:4826 , piece_id=ICX4E7GB2PYOJBILF3754FAL45TBSECQ6RTAH4PVDKM3THPKLZUA, seg_pos={0 0}, upload=716]: Done uploading piece. err: ecclient: upload failed (node:13KQwy5RDTnWanKbCdnsdqRt88m58bEqcLmT3SpLxYW2uANXGv, address:104.36.82.42:4826): protocol: write tcp 159.223.143.39:59824->104.36.82.42:4826: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:39:08.795868246 (47m24.576801635s) [seg_pos={0 0}, upload=716]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:39:08.820571335 (47m24.601504707s) [upload=716]: Done uploading segments... Jul 26 22:39:08.820612436 (47m24.601545814s) [upload=716]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.027940729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......XXX.XXX.X.XX..X..X......X.............X..............] duration: 473.050039ms FAIL (2025-07-26 22:38:59 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 473.082686ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 474.399669ms FAIL (2025-07-26 22:38:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 474.685858ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..............................X........X.X.X...XX...........] duration: 3.478931534s FAIL (2025-07-26 22:38:55 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478960013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 443.537956ms FAIL (2025-07-26 22:38:34 UTC) orbiter warning: uplink: too many requests Duration: 443.562905ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...XX..XXX..X..XX..X..XX.X....X....X........X...............] duration: 413.935643ms FAIL (2025-07-26 22:38:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 413.970188ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)