Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X....X....X........X......................................] duration: 519.204331ms FAIL (2025-07-26 22:55:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 519.238337ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....XX] duration: 1m0.01703931s FAIL (2025-07-26 22:55:32 UTC) ======ERROR====== uplink: context deadline exceeded ======LOGS====== gment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:54:32.438371519 (1h2m48.219304899s) [upload=955]: Done uploading segments... Jul 26 22:54:32.438402298 (1h2m48.219335675s) [upload=955]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:54:32.548997773 (1h2m48.329931151s) [upload=971]: Starting upload Jul 26 22:54:32.549112830 (1h2m48.330046211s) [upload=971]: Done starting upload Jul 26 22:54:32.549220419 (1h2m48.330153799s) [upload=971]: Uploading segments... Jul 26 22:54:32.549570455 (1h2m48.33050384s) [upload=971]: Got next segment. Inline: false Jul 26 22:54:32.549633930 (1h2m48.33056731s) [upload=971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:54:32.549640869 (1h2m48.330574247s) [upload=971]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:54:32.597154071 (1h2m48.37808744s) [seg_pos={0 0}, upload=971]: Begin upload segment... Jul 26 22:54:32.597256302 (1h2m48.378189667s) [seg_pos={0 0}, upload=971]: Done begin upload segment. Jul 26 22:54:32.597289071 (1h2m48.378222452s) [noise=true , address=45.91.116.8:4057 , piece_id=FCDXB7XKJYCZ3HFBISW5KLGGWN57S7XFBBPUHEQJEVNVVLY7CZ3A, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.597325403 (1h2m48.378258794s) [noise=true , address=79.127.205.234:4605 , piece_id=E2GCBR5YSDUD4EZFLSF6GWVU6H7ZXWSQQTDQRO3C5HDCTOZ6FLFA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.597826029 (1h2m48.378759419s) [noise=true , address=77.90.31.21:4885 , piece_id=KEI7M7TMNBHZYEDH3IK2KUXCONNYTDZEXZIFOQ4HOYMV7E4LITTQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.597837835 (1h2m48.3787712s) [noise=true , address=77.90.31.11:4585 , piece_id=DQKVLVHJ4LOSIKDY73DVPDU2DNKCINTGZ4E7ZXN5YWE2LWSHMYMQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.598279837 (1h2m48.379213226s) [noise=true , address=79.127.201.215:4359 , piece_id=BZPNWTVQDFBF4QQM2JBT5RULPCHKCCKRDD365EVQZVMQJCIQIIDA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.598297728 (1h2m48.379231111s) [noise=true , address=64.49.13.140:4508 , piece_id=5PTUXJK6LL3GHBIJROFHT4RXJXBINNQC3M662JSKJLUTEQDCGWEA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.598723772 (1h2m48.379657152s) [noise=true , address=45.91.116.14:4194 , piece_id=EQUU5VQL7BSI6IS5KK4JLLA6DRP4O6BJYVWMAHA24RH4DJNDAO3Q, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.598739114 (1h2m48.379672503s) [noise=true , address=79.127.220.97:4361 , piece_id=WQ3QJTHWC5ALCQZNGO7FB45HRRZW55XHKRTQRXHQFB7NJ7Y73F7A, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.599172099 (1h2m48.380105485s) [noise=true , address=45.91.116.12:4229 , piece_id=HXNEO7TB6ZA2ALJAX25WTT4DRSQBKGDVRY57WNGZT6CB4LGB3CMA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.599238563 (1h2m48.380171966s) [noise=true , address=79.127.219.47:4048 , piece_id=UC37CIEOV35UCVM5MHFH23R5OFN47SLYKZMXRP66JHHNBE5D3OTA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.599631938 (1h2m48.380565321s) [noise=true , address=79.127.201.215:4346 , piece_id=AFCOK2MOOS4ZTPLHWFICB5KHE7PZKMIXK2J3RAPSUUDLUSSO5RBQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.599653524 (1h2m48.380586914s) [noise=true , address=45.91.116.9:4267 , piece_id=OE57ZC2HOABRYXKTQSFTOVQ7TQ7VHUSITVOMMFKGMPHSQ3XCRPHQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.600093317 (1h2m48.381026707s) [noise=true , address=79.127.205.245:4244 , piece_id=EYN5ZWSWAXAKOGVRJHJXBESOPUG36M7SQXV6ZGB7LCLPDHU6JYNQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.600119131 (1h2m48.381052509s) [noise=true , address=77.90.31.13:4644 , piece_id=Y4D4T3SFZTF7NJJWPOMJLHOK23CVC7ZL3GSLYSNK2DQZVCTNXMQQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.600511137 (1h2m48.381444525s) [noise=true , address=64.49.13.138:4483 , piece_id=E6KQ3CMVTHOZ4YFXGA4KERL6JMQFWLQXSDV77E26TROURWNEVSGA, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.600544340 (1h2m48.381477722s) [noise=true , address=77.90.31.10:4533 , piece_id=CB2PNWRK4WGQD26LNCOQZYHM4PCHPKQZSV7GP7QSFZ4GDGBBK2FQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.600948113 (1h2m48.381881495s) [noise=true , address=79.127.205.246:4288 , piece_id=5FB3F4PGGRAWPFZJVOQVXOZ7DOGP4BX4A4725GFJGBYTI27EBT6A, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.602477077 (1h2m48.38341044s) [noise=true , address=156.146.43.225:4434 , piece_id=45W53IHFQDARAUDQYHPRTAVATANBAKXAH7F7MOVVZZH3DYUGQLSQ, seg_pos={0 0}, upload=971]: Uploading piece... Jul 26 22:54:32.714954727 (1h2m48.495888095s) [upload=971]: Next returned nil segment Jul 26 22:54:32.714976186 (1h2m48.495909574s) [upload=971]: Waiting for error group managing segments... Jul 26 22:54:32.734167577 (1h2m48.515100939s) [noise=true , address=77.90.31.11:4585 , piece_id=DQKVLVHJ4LOSIKDY73DVPDU2DNKCINTGZ4E7ZXN5YWE2LWSHMYMQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:32.750333100 (1h2m48.53126648s) [noise=true , address=77.90.31.21:4885 , piece_id=KEI7M7TMNBHZYEDH3IK2KUXCONNYTDZEXZIFOQ4HOYMV7E4LITTQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:33.093380297 (1h2m48.874313681s) [noise=true , address=64.49.13.138:4483 , piece_id=E6KQ3CMVTHOZ4YFXGA4KERL6JMQFWLQXSDV77E26TROURWNEVSGA, seg_pos={0 0}, upload=971]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:33.207266246 (1h2m48.988199618s) [noise=true , address=79.127.205.234:4605 , piece_id=E2GCBR5YSDUD4EZFLSF6GWVU6H7ZXWSQQTDQRO3C5HDCTOZ6FLFA, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:33.219489629 (1h2m49.000423012s) [noise=true , address=79.127.205.245:4244 , piece_id=EYN5ZWSWAXAKOGVRJHJXBESOPUG36M7SQXV6ZGB7LCLPDHU6JYNQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:33.220328517 (1h2m49.001261894s) [noise=true , address=79.127.205.246:4288 , piece_id=5FB3F4PGGRAWPFZJVOQVXOZ7DOGP4BX4A4725GFJGBYTI27EBT6A, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:33.701703946 (1h2m49.482637328s) [noise=true , address=79.127.201.215:4346 , piece_id=AFCOK2MOOS4ZTPLHWFICB5KHE7PZKMIXK2J3RAPSUUDLUSSO5RBQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:34.083112828 (1h2m49.86404622s) [noise=true , address=45.91.116.12:4229 , piece_id=HXNEO7TB6ZA2ALJAX25WTT4DRSQBKGDVRY57WNGZT6CB4LGB3CMA, seg_pos={0 0}, upload=971]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:35.969111312 (1h2m51.750044697s) [noise=true , address=77.90.31.10:4533 , piece_id=CB2PNWRK4WGQD26LNCOQZYHM4PCHPKQZSV7GP7QSFZ4GDGBBK2FQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:36.078077811 (1h2m51.859011176s) [noise=true , address=79.127.219.47:4048 , piece_id=UC37CIEOV35UCVM5MHFH23R5OFN47SLYKZMXRP66JHHNBE5D3OTA, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:36.328866515 (1h2m52.1097999s) [noise=true , address=64.49.13.140:4508 , piece_id=5PTUXJK6LL3GHBIJROFHT4RXJXBINNQC3M662JSKJLUTEQDCGWEA, seg_pos={0 0}, upload=971]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:36.777681573 (1h2m52.55861494s) [noise=true , address=156.146.43.225:4434 , piece_id=45W53IHFQDARAUDQYHPRTAVATANBAKXAH7F7MOVVZZH3DYUGQLSQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:36.877544518 (1h2m52.658477895s) [noise=true , address=77.90.31.13:4644 , piece_id=Y4D4T3SFZTF7NJJWPOMJLHOK23CVC7ZL3GSLYSNK2DQZVCTNXMQQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:37.912569159 (1h2m53.693502534s) [noise=true , address=79.127.220.97:4361 , piece_id=WQ3QJTHWC5ALCQZNGO7FB45HRRZW55XHKRTQRXHQFB7NJ7Y73F7A, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:54:41.263793146 (1h2m57.044726532s) [noise=true , address=45.91.116.9:4267 , piece_id=OE57ZC2HOABRYXKTQSFTOVQ7TQ7VHUSITVOMMFKGMPHSQ3XCRPHQ, seg_pos={0 0}, upload=971]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:51.099651867 (1h3m6.880585234s) [noise=true , address=45.91.116.14:4194 , piece_id=EQUU5VQL7BSI6IS5KK4JLLA6DRP4O6BJYVWMAHA24RH4DJNDAO3Q, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:55:04.514308623 (1h3m20.295242001s) [noise=true , address=45.91.116.8:4057 , piece_id=FCDXB7XKJYCZ3HFBISW5KLGGWN57S7XFBBPUHEQJEVNVVLY7CZ3A, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:55:27.566458463 (1h3m43.347391842s) [noise=true , address=79.127.201.215:4359 , piece_id=BZPNWTVQDFBF4QQM2JBT5RULPCHKCCKRDD365EVQZVMQJCIQIIDA, seg_pos={0 0}, upload=971]: Done uploading piece. err: <nil> Jul 26 22:55:27.566510882 (1h3m43.347444265s) [seg_pos={0 0}, upload=971]: Segment reached optimal threshold of 14 pieces. Jul 26 22:55:27.566618049 (1h3m43.347551427s) [seg_pos={0 0}, upload=971]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 26 22:55:27.566754342 (1h3m43.347687726s) [upload=971]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:55:27.566760421 (1h3m43.347693803s) [upload=971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:55:27.625705218 (1h3m43.406638598s) [upload=971]: Done uploading segments... Jul 26 22:55:27.625741607 (1h3m43.406674975s) [upload=971]: Upload finished. err: <nil> Duration: 1m0.017062051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 191.082476ms FAIL (2025-07-26 22:55:27 UTC) orbiter warning: uplink: too many requests Duration: 191.11263ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X......X.............X................X................] duration: 853.035021ms FAIL (2025-07-26 22:55:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 853.123775ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..........X........X.X.X...XX.................X.X..........X] duration: 1m0.002779866s FAIL (2025-07-26 22:55:07 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002830858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX.X....X....X........X.....................................] duration: 385.854916ms FAIL (2025-07-26 22:54:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 385.900514ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 254.63483ms FAIL (2025-07-26 22:54:46 UTC) orbiter warning: uplink: too many requests Duration: 254.687599ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....X] duration: 1m0.033336885s FAIL (2025-07-26 22:54:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:32.813119492 (1h1m48.594052867s) [noise=true , address=45.91.116.14:4222 , piece_id=EGKSRBFK5WDNEMV3MLLJQHDYC4DLPBYO6B2Q4HUXQII4HCADSIIQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:32.915679332 (1h1m48.696612717s) [noise=true , address=64.49.13.170:4718 , piece_id=L7US2PVTOTPT4BN5EFUZF6M46LSMRAPZANBETGW2UU4EKR6F5N2A, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:12AcLsSmVz9sRtuDLs47WoWYC66YSbr2aXBjzBdjayg22daQ2Hp, address:64.49.13.170:4718): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:33.091484827 (1h1m48.872418207s) [noise=true , address=79.127.205.234:4617 , piece_id=U3VUFBGNHTM4QW534E4ZEAHUU2ALSAQH4WTNHFD7BFW4NPSQXUQA, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:33.103315699 (1h1m48.884249065s) [noise=true , address=79.127.205.233:4650 , piece_id=3MPBPF5OFZ2WNEYTFBK5XIXPPOKBRMAEH6NXX7NJFA2WK2VKW3HA, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:33.123562195 (1h1m48.904495565s) [noise=true , address=79.127.205.235:4678 , piece_id=NVTCFEKIYNNX5THIUP5F2T53QXMTJJ7UGX2SMNQE4L7FN7DYHT7Q, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:33.123610490 (1h1m48.904543872s) [noise=true , address=79.127.205.246:4269 , piece_id=PZN3EWSFH3T2FOL53ZR2BIHYAGNFWJO46BMBHVJVJ3TN4NBFSKWQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:33.179739024 (1h1m48.960672388s) [noise=true , address=45.91.116.9:4267 , piece_id=F2HYXNNHKK2TERXV6IMLGMV6WYOBM3PG6RZFXABUCPTB6ZU6PBNQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:33.302600826 (1h1m49.083534205s) [noise=true , address=79.127.219.51:4210 , piece_id=2VCMGRQ7Z3LFEDUXBZR6SGOEGWRRGD6WWFTGAE4OETO4FPD5ESYA, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:33.398568410 (1h1m49.179501787s) [noise=true , address=64.49.13.138:4482 , piece_id=OE3QVTYPBVNAZPWTLDEQD3TRGTRBLXXU3RUQYV5BEWPGJQG6AQUA, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:35.779566556 (1h1m51.560499922s) [noise=true , address=156.146.43.225:4434 , piece_id=KXLKD5TPW3UVUX6RQXJYK4ODSTEUS7UZUN2DMSFSQBCUAL6NQP4Q, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:39.616191602 (1h1m55.39712498s) [noise=true , address=64.49.13.155:4916 , piece_id=HIGJ36DKG5LEXQ4OWI3PIKXBQFKTZH4VFXPVXASFMB5QSWXWZBXQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:53:39.926132760 (1h1m55.707066125s) [noise=true , address=45.91.116.8:4065 , piece_id=5ZCTIVA4ZF4P4AQ6KG5MTBC7C66HIAWMCUUFKUQT2DBA55KNGH3A, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:17EPCQmYb816aegE9vPSoJYByMUEvhP4DUiabis2gh9QjXQC3s, address:45.91.116.8:4065): protocol: storage node overloaded, request limit: 2000 Jul 26 22:53:44.275335248 (1h2m0.056268618s) [noise=true , address=45.91.116.14:4206 , piece_id=ZAMVAXIDZ5WMTGWHUL7QS7EDM3MVRQT74NKG45BBIWSXIKGOAEDQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:05.973978655 (1h2m21.754912044s) [noise=true , address=77.90.31.10:4525 , piece_id=U2HFJAXWHGK4SBE7RT2DH4KE3QYHUCMLORQMTFHS5IA5YOJWYX4A, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:12.504493203 (1h2m28.285426588s) [noise=true , address=77.90.31.10:4533 , piece_id=LQ7JRQT3CXMOVK47Z2V5AX6IA3KNGB55PN6FMTOS24M5AMFNM2RQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:12.525194370 (1h2m28.30612776s) [noise=true , address=79.127.205.235:4686 , piece_id=QQ2Z2JLEYTGWFTR2KRQDIJRLMZVX2PBSD4UJPFVU5HT57GLCQNLQ, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.525655577 (1h2m28.306588966s) [noise=true , address=156.146.43.225:4434 , piece_id=4U6GSBRIV2BFML7JAFUFQ7DVCVTWBN4J7U3LPZFFNVX2OFEV6Y5A, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.525977780 (1h2m28.306911168s) [noise=true , address=77.90.31.13:4653 , piece_id=Z75RQ2ALZAPWI4LFDWJHQZLPTXA5JFG4LSPRAONIZXMOFXYODCRQ, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.526116442 (1h2m28.307049829s) [noise=true , address=79.127.205.246:4272 , piece_id=WDZMFL2I43UNK4KTLJOAAGAFYPR2WXIV6IMFGDTTQCJDH22LLDLQ, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.527131188 (1h2m28.308064575s) [noise=true , address=79.127.219.38:4010 , piece_id=NIKB2G6PVUVZNHDHDPD4STPVJZZBTEJLQOWZ32VEBEOW7CVD2UMA, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.527195999 (1h2m28.308129384s) [noise=true , address=79.127.220.97:4361 , piece_id=FVZ7OJACA2AXDXMU4NZTJNQBJV73H2LR3EQM42VEGNUAS4BSIEYA, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.528268214 (1h2m28.309201604s) [noise=true , address=64.49.13.138:4483 , piece_id=4CXJ3NQWDCW5YHOOBQDX423FB4LGZKAHCQM5ZAZMLIOAUTKWQGSQ, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.528347124 (1h2m28.309280511s) [noise=true , address=64.49.13.140:4508 , piece_id=RJJHI2AZHBL5DGAQDHIZRYHFFGFKTFOALU37JAH7DGGR3MTPTYDA, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.529864133 (1h2m28.31079752s) [noise=true , address=45.91.116.14:4194 , piece_id=3XFIUZE74HU4ACKIEWC5TMZYOG2456UTWWBZI5VOAV7Y5ROTRYLA, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.530303305 (1h2m28.311236696s) [noise=true , address=79.127.201.215:4353 , piece_id=SAZFEY3MHKXZEDBCXRSKOHOCN4L5KBGCGUEBXEMVOY7S2OIYAG7Q, seg_pos={0 0}, upload=955]: Uploading piece... Jul 26 22:54:12.565693539 (1h2m28.34662692s) [noise=true , address=77.90.31.13:4653 , piece_id=Z75RQ2ALZAPWI4LFDWJHQZLPTXA5JFG4LSPRAONIZXMOFXYODCRQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, address:77.90.31.13:4653): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:12.583435196 (1h2m28.364368579s) [noise=true , address=79.127.219.38:4010 , piece_id=NIKB2G6PVUVZNHDHDPD4STPVJZZBTEJLQOWZ32VEBEOW7CVD2UMA, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:12.635341566 (1h2m28.416274935s) [noise=true , address=156.146.43.225:4434 , piece_id=4U6GSBRIV2BFML7JAFUFQ7DVCVTWBN4J7U3LPZFFNVX2OFEV6Y5A, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: expected piece hash; used serial already exists in store; EOF Jul 26 22:54:12.730251091 (1h2m28.511184471s) [noise=true , address=45.91.116.14:4194 , piece_id=3XFIUZE74HU4ACKIEWC5TMZYOG2456UTWWBZI5VOAV7Y5ROTRYLA, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:13.115781374 (1h2m28.896714739s) [noise=true , address=79.127.205.246:4272 , piece_id=WDZMFL2I43UNK4KTLJOAAGAFYPR2WXIV6IMFGDTTQCJDH22LLDLQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:13.121922344 (1h2m28.90285572s) [noise=true , address=79.127.205.235:4686 , piece_id=QQ2Z2JLEYTGWFTR2KRQDIJRLMZVX2PBSD4UJPFVU5HT57GLCQNLQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:13.159472054 (1h2m28.94040542s) [noise=true , address=79.127.220.97:4361 , piece_id=FVZ7OJACA2AXDXMU4NZTJNQBJV73H2LR3EQM42VEGNUAS4BSIEYA, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:13.191993414 (1h2m28.972926801s) [noise=true , address=64.49.13.140:4508 , piece_id=RJJHI2AZHBL5DGAQDHIZRYHFFGFKTFOALU37JAH7DGGR3MTPTYDA, seg_pos={0 0}, upload=955]: Done uploading piece. err: <nil> Jul 26 22:54:16.118043660 (1h2m31.898977036s) [noise=true , address=64.49.13.138:4483 , piece_id=4CXJ3NQWDCW5YHOOBQDX423FB4LGZKAHCQM5ZAZMLIOAUTKWQGSQ, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:54:32.408607043 (1h2m48.189540424s) [noise=true , address=79.127.201.215:4353 , piece_id=SAZFEY3MHKXZEDBCXRSKOHOCN4L5KBGCGUEBXEMVOY7S2OIYAG7Q, seg_pos={0 0}, upload=955]: Done uploading piece. err: ecclient: upload failed (node:1227Uk4K8hmXrae9npcgjziCNhwfdfdDPdgEryAy5beiFaSmd1W, address:79.127.201.215:4353): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:54:32.408658110 (1h2m48.189591498s) [seg_pos={0 0}, upload=955]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:54:32.438371519 (1h2m48.219304899s) [upload=955]: Done uploading segments... Jul 26 22:54:32.438402298 (1h2m48.219335675s) [upload=955]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.033398337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X..X......X.............X................X...............] duration: 429.058131ms FAIL (2025-07-26 22:54:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 429.096805ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 406.533933ms FAIL (2025-07-26 22:53:59 UTC) orbiter warning: uplink: too many requests Duration: 406.563498ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX.X....X....X........X....................................] duration: 735.398417ms FAIL (2025-07-26 22:53:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 735.428693ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........X........X.X.X...XX.................X.X..........] duration: 14.101728331s FAIL (2025-07-26 22:53:34 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.101767008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX....] duration: 30.790620249s FAIL (2025-07-26 22:53:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== pb.BatchRequestItem_SegmentBegin Jul 26 22:49:01.022497827 (57m16.803431216s) [seg_pos={0 0}, upload=881]: Begin upload segment... Jul 26 22:49:01.022580489 (57m16.803513879s) [seg_pos={0 0}, upload=881]: Done begin upload segment. Jul 26 22:49:01.022617556 (57m16.803550944s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023291722 (57m16.804225102s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023899131 (57m16.804832515s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024446671 (57m16.805380054s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024966954 (57m16.805900344s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.025508416 (57m16.806441807s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026008754 (57m16.806942141s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026427269 (57m16.80736065s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026853310 (57m16.807786706s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027286217 (57m16.808219607s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027718535 (57m16.808651921s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.029646196 (57m16.810579586s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030101602 (57m16.811034994s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030548215 (57m16.811481598s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030961321 (57m16.811894709s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031462001 (57m16.812395386s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031844379 (57m16.812777746s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.032259287 (57m16.813192676s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.102276736 (57m16.883210122s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.102735014 (57m16.883668406s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.123020337 (57m16.903953719s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170593572 (57m16.951526952s) [upload=881]: Next returned nil segment Jul 26 22:49:01.170610860 (57m16.951544242s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170633877 (57m16.951567243s) [upload=881]: Waiting for error group managing segments... Jul 26 22:49:01.195133094 (57m16.976066474s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.202867381 (57m16.98380075s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.226760035 (57m17.007693403s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.229442519 (57m17.010375908s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.715130515 (57m17.496063893s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.728479686 (57m17.509413058s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.196712183 (57m20.97764556s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.652598133 (57m21.433531515s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:06.470679063 (57m22.25161244s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:07.317498912 (57m23.098432629s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.623872606 (57m24.404805973s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1mY7sRFAeDLmKPqq8BgwnGtkdPTsDRR6soTBshsn4qhP7evARb, address:64.49.13.140:4515): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.736598453 (57m24.51753183s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:52.335446351 (58m8.116379721s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:50:00.909179814 (58m16.690113181s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:50:00.909239469 (58m16.690172853s) [seg_pos={0 0}, upload=881]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:50:00.935905580 (58m16.716838959s) [upload=881]: Done uploading segments... Jul 26 22:50:00.935961200 (58m16.716894583s) [upload=881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 30.790658883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX..X..X......X.............X................X..............] duration: 616.19925ms FAIL (2025-07-26 22:53:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 616.250881ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 194.542368ms FAIL (2025-07-26 22:52:58 UTC) orbiter warning: uplink: too many requests Duration: 194.586681ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX.X....X....X........X...................................] duration: 474.93512ms FAIL (2025-07-26 22:52:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 474.974744ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..XX.X....X....X........X..................................] duration: 580.006888ms FAIL (2025-07-26 22:52:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 580.043055ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [............X........X.X.X...XX.................X.X.........] duration: 8.641585244s FAIL (2025-07-26 22:52:19 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.641604382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX...] duration: 12.4304335s FAIL (2025-07-26 22:52:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== pb.BatchRequestItem_SegmentBegin Jul 26 22:49:01.022497827 (57m16.803431216s) [seg_pos={0 0}, upload=881]: Begin upload segment... Jul 26 22:49:01.022580489 (57m16.803513879s) [seg_pos={0 0}, upload=881]: Done begin upload segment. Jul 26 22:49:01.022617556 (57m16.803550944s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023291722 (57m16.804225102s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023899131 (57m16.804832515s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024446671 (57m16.805380054s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024966954 (57m16.805900344s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.025508416 (57m16.806441807s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026008754 (57m16.806942141s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026427269 (57m16.80736065s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026853310 (57m16.807786706s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027286217 (57m16.808219607s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027718535 (57m16.808651921s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.029646196 (57m16.810579586s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030101602 (57m16.811034994s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030548215 (57m16.811481598s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030961321 (57m16.811894709s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031462001 (57m16.812395386s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031844379 (57m16.812777746s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.032259287 (57m16.813192676s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.102276736 (57m16.883210122s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.102735014 (57m16.883668406s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.123020337 (57m16.903953719s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170593572 (57m16.951526952s) [upload=881]: Next returned nil segment Jul 26 22:49:01.170610860 (57m16.951544242s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170633877 (57m16.951567243s) [upload=881]: Waiting for error group managing segments... Jul 26 22:49:01.195133094 (57m16.976066474s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.202867381 (57m16.98380075s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.226760035 (57m17.007693403s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.229442519 (57m17.010375908s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.715130515 (57m17.496063893s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.728479686 (57m17.509413058s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.196712183 (57m20.97764556s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.652598133 (57m21.433531515s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:06.470679063 (57m22.25161244s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:07.317498912 (57m23.098432629s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.623872606 (57m24.404805973s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1mY7sRFAeDLmKPqq8BgwnGtkdPTsDRR6soTBshsn4qhP7evARb, address:64.49.13.140:4515): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.736598453 (57m24.51753183s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:52.335446351 (58m8.116379721s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:50:00.909179814 (58m16.690113181s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:50:00.909239469 (58m16.690172853s) [seg_pos={0 0}, upload=881]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:50:00.935905580 (58m16.716838959s) [upload=881]: Done uploading segments... Jul 26 22:50:00.935961200 (58m16.716894583s) [upload=881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 12.43047775s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XX..X..X......X.............X................X.............] duration: 474.319762ms FAIL (2025-07-26 22:52:07 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 474.353377ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 182.60876ms FAIL (2025-07-26 22:51:51 UTC) orbiter warning: uplink: too many requests Duration: 182.648446ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.XX..X..X......X.............X................X............] duration: 4.928103077s FAIL (2025-07-26 22:51:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.928141149s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..XX.X....X....X........X.................................] duration: 556.65238ms FAIL (2025-07-26 22:51:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 556.684713ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX..] duration: 40.863851211s FAIL (2025-07-26 22:51:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== pb.BatchRequestItem_SegmentBegin Jul 26 22:49:01.022497827 (57m16.803431216s) [seg_pos={0 0}, upload=881]: Begin upload segment... Jul 26 22:49:01.022580489 (57m16.803513879s) [seg_pos={0 0}, upload=881]: Done begin upload segment. Jul 26 22:49:01.022617556 (57m16.803550944s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023291722 (57m16.804225102s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023899131 (57m16.804832515s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024446671 (57m16.805380054s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024966954 (57m16.805900344s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.025508416 (57m16.806441807s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026008754 (57m16.806942141s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026427269 (57m16.80736065s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026853310 (57m16.807786706s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027286217 (57m16.808219607s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027718535 (57m16.808651921s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.029646196 (57m16.810579586s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030101602 (57m16.811034994s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030548215 (57m16.811481598s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030961321 (57m16.811894709s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031462001 (57m16.812395386s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031844379 (57m16.812777746s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.032259287 (57m16.813192676s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.102276736 (57m16.883210122s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.102735014 (57m16.883668406s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.123020337 (57m16.903953719s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170593572 (57m16.951526952s) [upload=881]: Next returned nil segment Jul 26 22:49:01.170610860 (57m16.951544242s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170633877 (57m16.951567243s) [upload=881]: Waiting for error group managing segments... Jul 26 22:49:01.195133094 (57m16.976066474s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.202867381 (57m16.98380075s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.226760035 (57m17.007693403s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.229442519 (57m17.010375908s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.715130515 (57m17.496063893s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.728479686 (57m17.509413058s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.196712183 (57m20.97764556s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.652598133 (57m21.433531515s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:06.470679063 (57m22.25161244s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:07.317498912 (57m23.098432629s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.623872606 (57m24.404805973s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1mY7sRFAeDLmKPqq8BgwnGtkdPTsDRR6soTBshsn4qhP7evARb, address:64.49.13.140:4515): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.736598453 (57m24.51753183s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:52.335446351 (58m8.116379721s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:50:00.909179814 (58m16.690113181s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:50:00.909239469 (58m16.690172853s) [seg_pos={0 0}, upload=881]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:50:00.935905580 (58m16.716838959s) [upload=881]: Done uploading segments... Jul 26 22:50:00.935961200 (58m16.716894583s) [upload=881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 40.863879706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X..XX.X....X....X........X................................] duration: 526.257908ms FAIL (2025-07-26 22:51:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 526.289938ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 219.731484ms FAIL (2025-07-26 22:51:24 UTC) orbiter warning: uplink: too many requests Duration: 219.756081ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.............X........X.X.X...XX.................X.X........] duration: 20.571982582s FAIL (2025-07-26 22:51:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.572012793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..X..XX.X....X....X........X...............................] duration: 690.084306ms FAIL (2025-07-26 22:51:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 690.12536ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.XX..X..X......X.............X................X...........] duration: 432.307049ms FAIL (2025-07-26 22:50:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 432.344366ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 193.261732ms FAIL (2025-07-26 22:50:43 UTC) orbiter warning: uplink: too many requests Duration: 193.289595ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..............X........X.X.X...XX.................X.X.......] duration: 14.771688482s FAIL (2025-07-26 22:50:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.771715349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX.] duration: 29.252017951s FAIL (2025-07-26 22:50:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== pb.BatchRequestItem_SegmentBegin Jul 26 22:49:01.022497827 (57m16.803431216s) [seg_pos={0 0}, upload=881]: Begin upload segment... Jul 26 22:49:01.022580489 (57m16.803513879s) [seg_pos={0 0}, upload=881]: Done begin upload segment. Jul 26 22:49:01.022617556 (57m16.803550944s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023291722 (57m16.804225102s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023899131 (57m16.804832515s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024446671 (57m16.805380054s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024966954 (57m16.805900344s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.025508416 (57m16.806441807s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026008754 (57m16.806942141s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026427269 (57m16.80736065s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026853310 (57m16.807786706s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027286217 (57m16.808219607s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027718535 (57m16.808651921s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.029646196 (57m16.810579586s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030101602 (57m16.811034994s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030548215 (57m16.811481598s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030961321 (57m16.811894709s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031462001 (57m16.812395386s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031844379 (57m16.812777746s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.032259287 (57m16.813192676s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.102276736 (57m16.883210122s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.102735014 (57m16.883668406s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.123020337 (57m16.903953719s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170593572 (57m16.951526952s) [upload=881]: Next returned nil segment Jul 26 22:49:01.170610860 (57m16.951544242s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170633877 (57m16.951567243s) [upload=881]: Waiting for error group managing segments... Jul 26 22:49:01.195133094 (57m16.976066474s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.202867381 (57m16.98380075s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.226760035 (57m17.007693403s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.229442519 (57m17.010375908s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.715130515 (57m17.496063893s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.728479686 (57m17.509413058s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.196712183 (57m20.97764556s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.652598133 (57m21.433531515s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:06.470679063 (57m22.25161244s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:07.317498912 (57m23.098432629s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.623872606 (57m24.404805973s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1mY7sRFAeDLmKPqq8BgwnGtkdPTsDRR6soTBshsn4qhP7evARb, address:64.49.13.140:4515): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.736598453 (57m24.51753183s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:52.335446351 (58m8.116379721s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:50:00.909179814 (58m16.690113181s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:50:00.909239469 (58m16.690172853s) [seg_pos={0 0}, upload=881]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:50:00.935905580 (58m16.716838959s) [upload=881]: Done uploading segments... Jul 26 22:50:00.935961200 (58m16.716894583s) [upload=881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 29.252044302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...............X........X.X.X...XX.................X.X......] duration: 9.42755031s FAIL (2025-07-26 22:50:15 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.427578201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX..X..XX.X....X....X........X..............................] duration: 691.723185ms FAIL (2025-07-26 22:50:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 691.76898ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.X.XX..X..X......X.............X................X..........] duration: 563.524153ms FAIL (2025-07-26 22:50:07 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 563.565295ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 194.418384ms FAIL (2025-07-26 22:50:07 UTC) orbiter warning: uplink: too many requests Duration: 194.45432ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.XX] duration: 1m0.030108985s FAIL (2025-07-26 22:50:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== pb.BatchRequestItem_SegmentBegin Jul 26 22:49:01.022497827 (57m16.803431216s) [seg_pos={0 0}, upload=881]: Begin upload segment... Jul 26 22:49:01.022580489 (57m16.803513879s) [seg_pos={0 0}, upload=881]: Done begin upload segment. Jul 26 22:49:01.022617556 (57m16.803550944s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023291722 (57m16.804225102s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.023899131 (57m16.804832515s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024446671 (57m16.805380054s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.024966954 (57m16.805900344s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.025508416 (57m16.806441807s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026008754 (57m16.806942141s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026427269 (57m16.80736065s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.026853310 (57m16.807786706s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027286217 (57m16.808219607s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.027718535 (57m16.808651921s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.029646196 (57m16.810579586s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030101602 (57m16.811034994s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030548215 (57m16.811481598s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.030961321 (57m16.811894709s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031462001 (57m16.812395386s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.031844379 (57m16.812777746s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.032259287 (57m16.813192676s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Uploading piece... Jul 26 22:49:01.102276736 (57m16.883210122s) [noise=true , address=77.90.31.13:4648 , piece_id=NG6G4EOTJC3ST7B4GGU5TGFGA275RTZITSTFDHP7KLQINYW5VPJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.102735014 (57m16.883668406s) [noise=true , address=79.127.219.51:4212 , piece_id=37GF4DHRKANEIZE32ZYY32VB6CL5FTNXG57EOVWNOUYXPDQT7VMA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.123020337 (57m16.903953719s) [noise=true , address=79.127.219.51:4210 , piece_id=7T6SWEZ456SILZCR6MKLFGQ4XDTLQXLY2LMAHGWXSOX2TGHYAIJA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170593572 (57m16.951526952s) [upload=881]: Next returned nil segment Jul 26 22:49:01.170610860 (57m16.951544242s) [noise=true , address=79.127.219.48:4090 , piece_id=GQWF36VUGF3CD3JK4ROEA2ZGLHRMRZH7VLLEGYRYNSPMJB3RYB4A, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.170633877 (57m16.951567243s) [upload=881]: Waiting for error group managing segments... Jul 26 22:49:01.195133094 (57m16.976066474s) [noise=true , address=79.127.220.101:4048 , piece_id=2MJC3AZFYOGWHWRM23BWECBESUG3R53JWVSUTQSDDKZJHDEFJS4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.202867381 (57m16.98380075s) [noise=true , address=77.90.31.19:4874 , piece_id=SHDNCSPNU3IYJNW4LG66T7ECGD3MNSISQEVOOA2KJUU2Q25AXVHQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.226760035 (57m17.007693403s) [noise=true , address=156.146.43.225:4434 , piece_id=C63OEPG5E7MZYGYIDEUUV6UPYLB2IRPON44OFVJ2KIUX7T3TTGQA, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.229442519 (57m17.010375908s) [noise=true , address=156.146.43.225:4448 , piece_id=H6WPBJGBTGYPNTZ54MJDUYCJ2OEUXXLNZJRZJT7WYD3EFTNTVBFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:01.715130515 (57m17.496063893s) [noise=true , address=79.127.205.246:4286 , piece_id=S5LSWRKB4BOQACXUVBPS4OSGFGHD37FLYZHBM7CCSDG47W6GF5BQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:01.728479686 (57m17.509413058s) [noise=true , address=79.127.205.245:4247 , piece_id=JXVT4VMJWEPG6RUU6A2MV7VN3UP5UO26CLWMVSORZOEOVIVL2WDQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.196712183 (57m20.97764556s) [noise=true , address=79.127.220.97:4361 , piece_id=E7EMOBYVGUX6PDFTEO5UVUYNO574TJOY7NH2T35TYYMARXPJM7UQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:05.652598133 (57m21.433531515s) [noise=true , address=79.127.219.47:4048 , piece_id=GLX3LFYDHFL4QDNWXUQJEP4H3KA7HOIFQP2AFTA3CUL5Q4RNVD4Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:06.470679063 (57m22.25161244s) [noise=true , address=193.149.164.143:4696 , piece_id=6FLW6N42TYHFEK7R5RH55ZXNF27KXJTYTXUX2LKWMTDUTNXJIIPA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:49:07.317498912 (57m23.098432629s) [noise=true , address=64.49.13.140:4508 , piece_id=ILJZGFK3YH2NG4OZXKUYQI7JFCTMJQLAWSXXYYWKIKWLQSOSWV6Q, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.623872606 (57m24.404805973s) [noise=true , address=64.49.13.140:4515 , piece_id=M3VL3T7QNZMYU26UY5Z4TIK2GD7LIFYSDUDG6SKNTMSPICHFAGEQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1mY7sRFAeDLmKPqq8BgwnGtkdPTsDRR6soTBshsn4qhP7evARb, address:64.49.13.140:4515): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:08.736598453 (57m24.51753183s) [noise=true , address=45.91.116.14:4206 , piece_id=3GQHES5VG4QDX52JNC5O3PGLBPQYZC5V6AOUR4FYPKFBSQIOBIFQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:52.335446351 (58m8.116379721s) [noise=true , address=79.127.201.215:4346 , piece_id=LABVGF6GSZGBWU22U7J4MREUPSC3GGYFIOS5WFM3NTL32BAT44QA, seg_pos={0 0}, upload=881]: Done uploading piece. err: <nil> Jul 26 22:50:00.909179814 (58m16.690113181s) [noise=true , address=77.90.31.15:4713 , piece_id=SJFK6OIWEHI5POEDQMARFSITK37JVX5XUOP43SOUSNC5CITLPMZQ, seg_pos={0 0}, upload=881]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:50:00.909239469 (58m16.690172853s) [seg_pos={0 0}, upload=881]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:50:00.935905580 (58m16.716838959s) [upload=881]: Done uploading segments... Jul 26 22:50:00.935961200 (58m16.716894583s) [upload=881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.030132136s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [................X........X.X.X...XX.................X.X.....] duration: 15.808078289s FAIL (2025-07-26 22:49:41 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.80810688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX..X..XX.X....X....X........X.............................] duration: 505.002991ms FAIL (2025-07-26 22:49:22 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 505.065407ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 515.320696ms FAIL (2025-07-26 22:49:14 UTC) orbiter warning: uplink: too many requests Duration: 515.370615ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.................X........X.X.X...XX.................X.X....] duration: 8.524631739s FAIL (2025-07-26 22:49:02 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.524661719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..XXX.XXX.XXX.X.X] duration: 1m0.030661436s FAIL (2025-07-26 22:49:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== upload=848]: Done uploading segments... Jul 26 22:47:06.031403958 (55m21.812337338s) [upload=848]: Upload finished. err: <nil> Jul 26 22:48:00.835865140 (56m16.616798528s) [upload=867]: Starting upload Jul 26 22:48:00.835965549 (56m16.616898933s) [upload=867]: Done starting upload Jul 26 22:48:00.836155926 (56m16.617089289s) [upload=867]: Uploading segments... Jul 26 22:48:00.836664376 (56m16.617597748s) [upload=867]: Got next segment. Inline: false Jul 26 22:48:00.836929002 (56m16.617862392s) [upload=867]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:48:00.836939938 (56m16.617873325s) [upload=867]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:48:00.868865570 (56m16.649798988s) [seg_pos={0 0}, upload=867]: Begin upload segment... Jul 26 22:48:00.868927070 (56m16.64986045s) [seg_pos={0 0}, upload=867]: Done begin upload segment. Jul 26 22:48:00.868968799 (56m16.649902183s) [noise=true , address=156.146.43.225:4448 , piece_id=BCQ5N3NYEYS6NCLY6LTWS6JPLBYHNVKNJR4PWHANQUGCS2GUBRNA, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.874201539 (56m16.655134927s) [noise=true , address=45.91.116.8:4065 , piece_id=6PPF3NMSDWEPLYZRDTJKT2DZCYCZTLH7S6ZVZH2RSSU7PKSTPX3Q, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.875158017 (56m16.656091405s) [noise=true , address=77.90.31.16:4747 , piece_id=3XKCGVSOHD26BPOULVUPS4ZCGXYGK3NHXJ67QA7ED5TUJNDBQOWQ, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.875859561 (56m16.656792949s) [noise=true , address=64.49.13.138:4482 , piece_id=NI6IRGBDN2DYEJVLZFC4VNQPF43RB7PK33C7MI7LYPB3Y62FAPHA, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.877211626 (56m16.658145024s) [noise=true , address=79.127.219.47:4048 , piece_id=KAUT65BEMYGOARPD7A7W4RKFFGMOTYLVXTPK2ITBDU5JWRNMLO6A, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.878913621 (56m16.65984701s) [noise=true , address=45.91.116.14:4222 , piece_id=AF5YNC7PRVTLVGJHHT6QEZ2DY53MPXBHENEJRX7W56IGVKDJVOTQ, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.879802061 (56m16.660735449s) [noise=true , address=79.127.205.233:4649 , piece_id=FUP33YON3MS2YJT5NUCWUDYTGVLM3HXZGVCB7P6WIFTGV6PPOJEA, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.889352230 (56m16.670285615s) [noise=true , address=79.127.219.48:4090 , piece_id=FCHDLCHZEDAS543B5WGI232MEQA2KFO56SYLW5K47CL3KWSJGGUQ, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.891112723 (56m16.672046109s) [noise=true , address=45.91.116.5:4310 , piece_id=6WIDC6LIAOQGMBSVTSRF4Z7BFTR43H6NU7HKDEAV7SZ3MLHQAKSA, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.892311572 (56m16.673244959s) [noise=true , address=77.90.31.16:4743 , piece_id=AOOFKPBSHZACWZTT7NTDPO3K2PLDHYYTCC3ETAHMA5N2CGY3ROAA, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.892931456 (56m16.673864846s) [noise=true , address=79.127.205.245:4248 , piece_id=RIQONJJ2EKZFRB6VBXA2V3NM5PQLYORAVCRJOITBUUUTQA5JJUMQ, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.893397983 (56m16.674331368s) [noise=true , address=79.127.201.215:4341 , piece_id=TXPCI7BO32MDW6VN4GG2635NLMU3EHB6AJH4S76UA6WG7IWIQNMQ, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.893862149 (56m16.67479554s) [noise=true , address=79.127.220.101:4048 , piece_id=OKJIZXKDKUAH5FMTHQVRMYCZVDW7HJ7DDJIHNDB4NPMPVTA5NPLA, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.900118288 (56m16.681051674s) [noise=true , address=45.91.116.9:4269 , piece_id=K7XOUCXB2XDYZZ7BGQT32LE4VUHKZDHWPYFR3AJP37SE2I7PWZ3Q, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.900182138 (56m16.68111552s) [noise=true , address=79.127.219.38:4015 , piece_id=SFGMCDCQRS5NJVJYDCZY67R63LPVEY3KWITREWDH5BUZTGSEROQA, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.900842206 (56m16.681775592s) [noise=true , address=77.90.31.21:4887 , piece_id=YZTK76ZJJMWIJIHKQDFQSODTRK7BKH7T3SDTBC7YKFKXGJTEHQGA, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.901377038 (56m16.682310426s) [noise=true , address=156.146.43.225:4449 , piece_id=EE7BPZV7THHBSXB3RNKNJ7DYDTV7XGAFCB3P3ZT3ZV3OZUS6OU6A, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:00.901889908 (56m16.682823294s) [noise=true , address=64.49.13.170:4719 , piece_id=A4CABUVQ5ZYTYMXYOVF4OZFBVHVOVRUGZ5RPRSZIRCMUFQUVEEYA, seg_pos={0 0}, upload=867]: Uploading piece... Jul 26 22:48:01.055093141 (56m16.836026523s) [upload=867]: Next returned nil segment Jul 26 22:48:01.055120803 (56m16.836054189s) [upload=867]: Waiting for error group managing segments... Jul 26 22:48:01.088685321 (56m16.869618708s) [noise=true , address=79.127.219.38:4015 , piece_id=SFGMCDCQRS5NJVJYDCZY67R63LPVEY3KWITREWDH5BUZTGSEROQA, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:48:01.099240101 (56m16.880173464s) [noise=true , address=77.90.31.21:4887 , piece_id=YZTK76ZJJMWIJIHKQDFQSODTRK7BKH7T3SDTBC7YKFKXGJTEHQGA, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:48:01.100882204 (56m16.881815586s) [noise=true , address=79.127.219.48:4090 , piece_id=FCHDLCHZEDAS543B5WGI232MEQA2KFO56SYLW5K47CL3KWSJGGUQ, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:48:01.142451809 (56m16.92338519s) [noise=true , address=156.146.43.225:4448 , piece_id=BCQ5N3NYEYS6NCLY6LTWS6JPLBYHNVKNJR4PWHANQUGCS2GUBRNA, seg_pos={0 0}, upload=867]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:01.147972540 (56m16.928905923s) [noise=true , address=156.146.43.225:4449 , piece_id=EE7BPZV7THHBSXB3RNKNJ7DYDTV7XGAFCB3P3ZT3ZV3OZUS6OU6A, seg_pos={0 0}, upload=867]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:01.540977770 (56m17.32191115s) [noise=true , address=64.49.13.138:4482 , piece_id=NI6IRGBDN2DYEJVLZFC4VNQPF43RB7PK33C7MI7LYPB3Y62FAPHA, seg_pos={0 0}, upload=867]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:01.549827365 (56m17.330760736s) [noise=true , address=79.127.205.245:4248 , piece_id=RIQONJJ2EKZFRB6VBXA2V3NM5PQLYORAVCRJOITBUUUTQA5JJUMQ, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:48:01.565247746 (56m17.346181114s) [noise=true , address=79.127.205.233:4649 , piece_id=FUP33YON3MS2YJT5NUCWUDYTGVLM3HXZGVCB7P6WIFTGV6PPOJEA, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:48:01.788392325 (56m17.569325713s) [noise=true , address=45.91.116.14:4222 , piece_id=AF5YNC7PRVTLVGJHHT6QEZ2DY53MPXBHENEJRX7W56IGVKDJVOTQ, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:48:02.037763731 (56m17.818697114s) [noise=true , address=64.49.13.170:4719 , piece_id=A4CABUVQ5ZYTYMXYOVF4OZFBVHVOVRUGZ5RPRSZIRCMUFQUVEEYA, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:48:02.070914450 (56m17.85184783s) [noise=true , address=79.127.201.215:4341 , piece_id=TXPCI7BO32MDW6VN4GG2635NLMU3EHB6AJH4S76UA6WG7IWIQNMQ, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:48:03.372941113 (56m19.153874493s) [noise=true , address=45.91.116.8:4065 , piece_id=6PPF3NMSDWEPLYZRDTJKT2DZCYCZTLH7S6ZVZH2RSSU7PKSTPX3Q, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:48:03.421036137 (56m19.20196952s) [noise=true , address=45.91.116.5:4310 , piece_id=6WIDC6LIAOQGMBSVTSRF4Z7BFTR43H6NU7HKDEAV7SZ3MLHQAKSA, seg_pos={0 0}, upload=867]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:04.221966942 (56m20.002900311s) [noise=true , address=79.127.219.47:4048 , piece_id=KAUT65BEMYGOARPD7A7W4RKFFGMOTYLVXTPK2ITBDU5JWRNMLO6A, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:48:04.573205912 (56m20.354139296s) [noise=true , address=45.91.116.9:4269 , piece_id=K7XOUCXB2XDYZZ7BGQT32LE4VUHKZDHWPYFR3AJP37SE2I7PWZ3Q, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:48:05.184944663 (56m20.965878047s) [noise=true , address=79.127.220.101:4048 , piece_id=OKJIZXKDKUAH5FMTHQVRMYCZVDW7HJ7DDJIHNDB4NPMPVTA5NPLA, seg_pos={0 0}, upload=867]: Done uploading piece. err: <nil> Jul 26 22:49:00.798928216 (57m16.579861597s) [noise=true , address=77.90.31.16:4747 , piece_id=3XKCGVSOHD26BPOULVUPS4ZCGXYGK3NHXJ67QA7ED5TUJNDBQOWQ, seg_pos={0 0}, upload=867]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: write tcp 159.223.143.39:52980->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:49:00.798979280 (57m16.579912649s) [noise=true , address=77.90.31.16:4743 , piece_id=AOOFKPBSHZACWZTT7NTDPO3K2PLDHYYTCC3ETAHMA5N2CGY3ROAA, seg_pos={0 0}, upload=867]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: rpc: dial tcp 77.90.31.16:4743: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:49:00.799010902 (57m16.579944266s) [seg_pos={0 0}, upload=867]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:49:00.826467955 (57m16.607401342s) [upload=867]: Done uploading segments... Jul 26 22:49:00.826502990 (57m16.60743638s) [upload=867]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.030686711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX.X.XX..X..X......X.............X................X.........] duration: 589.914228ms FAIL (2025-07-26 22:48:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 590.016211ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XX..X..XX.X....X....X........X............................] duration: 479.999786ms FAIL (2025-07-26 22:48:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 480.0334ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 593.007182ms FAIL (2025-07-26 22:48:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 593.056391ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXX.X.XX..X..X......X.............X................X........] duration: 466.432412ms FAIL (2025-07-26 22:48:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 466.466625ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 222.705098ms FAIL (2025-07-26 22:48:11 UTC) orbiter warning: uplink: too many requests Duration: 222.737452ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..................X........X.X.X...XX.................X.X...] duration: 7.292616995s FAIL (2025-07-26 22:48:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.292647539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 530.141468ms FAIL (2025-07-26 22:47:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 530.220936ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X..XX..X..XX.X....X....X........X...........................] duration: 383.495013ms FAIL (2025-07-26 22:47:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 383.526701ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)