Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..X] duration: 30.032623612s FAIL (2025-07-27 00:31:42 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 (2h38m27.729427932s) [noise=true , address=104.36.82.2:4732 , piece_id=FMVOOI66OYTQ5MMLQJDKV33SH5UBOBGH4XVSOIPXUZPHO5FIG7MQ, seg_pos={0 0}, upload=2433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 159.223.143.39:54996->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:11.948517990 (2h38m27.729451357s) [seg_pos={0 0}, upload=2433]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:30:11.948618962 (2h38m27.729552332s) [upload=2433]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:30:11.948626030 (2h38m27.729559393s) [upload=2433]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:30:12.018855324 (2h38m27.799788716s) [upload=2433]: Done uploading segments... Jul 27 00:30:12.018897650 (2h38m27.799831049s) [upload=2433]: Upload finished. err: <nil> Jul 27 00:31:12.159695723 (2h39m27.940629087s) [upload=2448]: Starting upload Jul 27 00:31:12.159767937 (2h39m27.940701312s) [upload=2448]: Done starting upload Jul 27 00:31:12.160068267 (2h39m27.94100163s) [upload=2448]: Uploading segments... Jul 27 00:31:12.160432583 (2h39m27.941365963s) [upload=2448]: Got next segment. Inline: false Jul 27 00:31:12.160520325 (2h39m27.941453712s) [upload=2448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:12.160546955 (2h39m27.941480339s) [upload=2448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:12.197153293 (2h39m27.978086678s) [seg_pos={0 0}, upload=2448]: Begin upload segment... Jul 27 00:31:12.197241485 (2h39m27.978174874s) [noise=true , address=45.91.116.10:4082 , piece_id=2IN4MTZPW6BQ6ZMBJYM45SEBNFMHKDAXKIAZQLJP6PSBQUJ24U2Q, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.202968821 (2h39m27.98390221s) [seg_pos={0 0}, upload=2448]: Done begin upload segment. Jul 27 00:31:12.203093298 (2h39m27.984026687s) [noise=true , address=77.90.31.14:4662 , piece_id=OU2W5SJZ7NU5ZTJQQHOIPN7LCEAQT5OGBW2CKJ4USKGWKDAYMFKA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.203579409 (2h39m27.984512799s) [noise=true , address=77.90.31.14:4663 , piece_id=FRZPEUEMNTJDX27WADLUBC22CXE74JPSFOUDNNL7NJ4TMCNZMF5A, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.204015604 (2h39m27.984948989s) [noise=true , address=45.91.116.12:4225 , piece_id=LFZYMP5GSIOZDCYFHPMY64PLVZC5CBCNK7DO2M77GCL4JKA46TNA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.204401073 (2h39m27.985334462s) [noise=true , address=77.90.31.13:4627 , piece_id=AZSM4QUF77TECDU77PFUKT6YALD6Y4PUJBTJSAHY7UYFX46WLQMA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.205013220 (2h39m27.985946607s) [noise=true , address=45.91.116.13:4155 , piece_id=3MJY3OVR34AU53LQFYLYCJ7IQVOGM3SKN6VRSCIZF6XHVDKJQI3Q, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.205444844 (2h39m27.986378234s) [noise=true , address=104.36.82.5:4839 , piece_id=FABXNVRH7NJS5V6OJ3EBQVXBIUTVYPBRQC6OXECMZANGYNWMNMZQ, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.205810422 (2h39m27.986743808s) [noise=true , address=45.91.116.5:4298 , piece_id=M6EQY4PDGWSZOC7SXMEJXT3FGKOH5KLJ4U4L5DECVHXX6HSKNAWA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.206184793 (2h39m27.987118178s) [noise=true , address=77.90.31.15:4698 , piece_id=E24T5XO47376KU65ACGW2VJIOQ57YXKZUN6YVQDPVCQNAG5RZH2A, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.206619846 (2h39m27.987553232s) [noise=true , address=45.91.116.5:4299 , piece_id=ODBKEKETYG7L3UKMXYTFJAINZIOHG4YGDUSXM6JKMBDYKI4VFEXA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.207165081 (2h39m27.988098471s) [noise=true , address=45.91.116.14:4191 , piece_id=PO2GPLLZU7CQ3AH7KK57TRRZPAPX26JB7B3O4XA3ZO3MNOMK4RKQ, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.207850036 (2h39m27.988783423s) [noise=true , address=104.36.82.13:4185 , piece_id=WRNHR7X33VHZMMDCLHBIJFXU3HYNJT5FLATA34JMOZFNIC5ATDLQ, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.209818931 (2h39m27.990752331s) [noise=true , address=45.91.116.6:4973 , piece_id=PJ6SFQ37O525LBTWMFVXPUTZODL43XO5O5X7PCCZ4ONKBZNTQL7Q, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.210267025 (2h39m27.991200412s) [noise=true , address=45.91.116.8:4045 , piece_id=4GLSLSJ4SEYP6Z4LUZKC3UUT44KEZXUJRTMGVR23DKLV5M3PIK5A, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.210661756 (2h39m27.991595142s) [noise=true , address=104.36.82.8:4969 , piece_id=CA667UCM3VDDIPWLRZ6URFSSJOJXMUAQX4KJWJSYLL3DJBXWK7CA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.214500450 (2h39m27.995433836s) [noise=true , address=77.90.31.11:4553 , piece_id=NFR6VI445RLIQ7AO6GCEMASE5MKGQYHMMRWJCCLNORO3C6YIXRXA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.322882780 (2h39m28.103816164s) [upload=2448]: Next returned nil segment Jul 27 00:31:12.322902688 (2h39m28.103836074s) [upload=2448]: Waiting for error group managing segments... Jul 27 00:31:12.327209057 (2h39m28.108142436s) [noise=true , address=45.91.116.5:4299 , piece_id=ODBKEKETYG7L3UKMXYTFJAINZIOHG4YGDUSXM6JKMBDYKI4VFEXA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.328683217 (2h39m28.109616595s) [noise=true , address=45.91.116.6:4973 , piece_id=PJ6SFQ37O525LBTWMFVXPUTZODL43XO5O5X7PCCZ4ONKBZNTQL7Q, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.329209607 (2h39m28.110142971s) [noise=true , address=45.91.116.12:4225 , piece_id=LFZYMP5GSIOZDCYFHPMY64PLVZC5CBCNK7DO2M77GCL4JKA46TNA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.330333791 (2h39m28.111267177s) [noise=true , address=77.90.31.14:4662 , piece_id=OU2W5SJZ7NU5ZTJQQHOIPN7LCEAQT5OGBW2CKJ4USKGWKDAYMFKA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.331153586 (2h39m28.112086957s) [noise=true , address=45.91.116.10:4082 , piece_id=2IN4MTZPW6BQ6ZMBJYM45SEBNFMHKDAXKIAZQLJP6PSBQUJ24U2Q, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.332133429 (2h39m28.113066797s) [noise=true , address=77.90.31.14:4663 , piece_id=FRZPEUEMNTJDX27WADLUBC22CXE74JPSFOUDNNL7NJ4TMCNZMF5A, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.332833107 (2h39m28.113766485s) [noise=true , address=45.91.116.8:4045 , piece_id=4GLSLSJ4SEYP6Z4LUZKC3UUT44KEZXUJRTMGVR23DKLV5M3PIK5A, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.333485917 (2h39m28.11441928s) [noise=true , address=45.91.116.5:4298 , piece_id=M6EQY4PDGWSZOC7SXMEJXT3FGKOH5KLJ4U4L5DECVHXX6HSKNAWA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.334923028 (2h39m28.115856408s) [noise=true , address=45.91.116.13:4155 , piece_id=3MJY3OVR34AU53LQFYLYCJ7IQVOGM3SKN6VRSCIZF6XHVDKJQI3Q, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.377488636 (2h39m28.158422001s) [noise=true , address=45.91.116.14:4191 , piece_id=PO2GPLLZU7CQ3AH7KK57TRRZPAPX26JB7B3O4XA3ZO3MNOMK4RKQ, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.442882082 (2h39m28.22381546s) [noise=true , address=77.90.31.11:4553 , piece_id=NFR6VI445RLIQ7AO6GCEMASE5MKGQYHMMRWJCCLNORO3C6YIXRXA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:13.842044043 (2h39m29.62297741s) [noise=true , address=77.90.31.13:4627 , piece_id=AZSM4QUF77TECDU77PFUKT6YALD6Y4PUJBTJSAHY7UYFX46WLQMA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:42.128926436 (2h39m57.909859814s) [noise=true , address=104.36.82.5:4839 , piece_id=FABXNVRH7NJS5V6OJ3EBQVXBIUTVYPBRQC6OXECMZANGYNWMNMZQ, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129053492 (2h39m57.909986871s) [noise=true , address=104.36.82.13:4185 , piece_id=WRNHR7X33VHZMMDCLHBIJFXU3HYNJT5FLATA34JMOZFNIC5ATDLQ, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129130004 (2h39m57.910063384s) [noise=true , address=104.36.82.8:4969 , piece_id=CA667UCM3VDDIPWLRZ6URFSSJOJXMUAQX4KJWJSYLL3DJBXWK7CA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129201387 (2h39m57.910134767s) [noise=true , address=77.90.31.15:4698 , piece_id=E24T5XO47376KU65ACGW2VJIOQ57YXKZUN6YVQDPVCQNAG5RZH2A, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 159.223.143.39:40516->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129222200 (2h39m57.91015558s) [seg_pos={0 0}, upload=2448]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:31:42.156334822 (2h39m57.937268207s) [upload=2448]: Done uploading segments... Jul 27 00:31:42.156370339 (2h39m57.937303719s) [upload=2448]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.032649518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 223.567255ms FAIL (2025-07-27 00:31:36 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 223.60094ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029963095s FAIL (2025-07-27 00:31:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Inline: false Jul 27 00:30:26.724536733 (2h38m42.505470119s) [upload=2437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:30:26.724543456 (2h38m42.505476837s) [upload=2437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:30:26.759088225 (2h38m42.540021593s) [seg_pos={0 0}, upload=2437]: Begin upload segment... Jul 27 00:30:26.759219093 (2h38m42.540152474s) [seg_pos={0 0}, upload=2437]: Done begin upload segment. Jul 27 00:30:26.759245235 (2h38m42.540178624s) [noise=true , address=77.90.31.21:4888 , piece_id=3EZ5Y5VMPSQJJDHKSBFYJPKPMTYNL2TIJMZLXE3KQNMN4IITFKIQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.759261652 (2h38m42.540195038s) [noise=true , address=77.90.31.12:4617 , piece_id=G5M3FQUB6XYM5JST2XDUWJLUT7AEQ445AGIS3C7UIJVUNF6SW2UQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.759324596 (2h38m42.540257959s) [noise=true , address=79.127.219.38:4003 , piece_id=HQMHFFMF5RTMY6745SMZMWONAUZHOC6BN22CFZ2TFV2PDQYPMULQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.760005812 (2h38m42.5409392s) [noise=true , address=79.127.220.97:4361 , piece_id=CFYELNCF25IV7UH7DZQ44FORQNPN3AUDYUWZTMI4Y2MKNQYZ3BJQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.760022149 (2h38m42.540955534s) [noise=true , address=156.146.43.225:4434 , piece_id=NR64SDAEX7AV2JSAPI53NHWJTCDS2Z4JUGBBNR7ZRLA4O56LMEGQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.760072046 (2h38m42.541005414s) [noise=true , address=104.36.82.46:4482 , piece_id=2ZDWBSBPM5ZYALK2LLM7ZS52CAF6SELSMGRFFQRXVGEWKDPCRJPA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.760523955 (2h38m42.541457343s) [noise=true , address=77.90.31.16:4759 , piece_id=YBKDJVPNOEJS7OES46CDLEGMEITIZXFEI4M72ZYQX27J4Q3MTOSQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.760583534 (2h38m42.541516922s) [noise=true , address=79.127.201.215:4349 , piece_id=MW2CVLRLZT3O3XETRD5XJXJX3QGURHTXRMSBST4MZ3KVVV34PCRQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.761023884 (2h38m42.541957271s) [noise=true , address=104.36.82.47:4542 , piece_id=YLHHFQFRQ777FOACE2ZWJ4W555GKMJ75QO4IITBVLCHM77235KBA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.761074731 (2h38m42.542008122s) [noise=true , address=77.90.31.13:4636 , piece_id=Q445H2SJ6ZEOI5GLWLPAWPYEYS6K4X5BP5HJBVSCDKK3ICEQF4BA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.761431108 (2h38m42.54236449s) [noise=true , address=77.90.31.19:4871 , piece_id=HGGMRVOTBRBKCKUW2KBIHVXJEQBVNKGT5XD7AMY4KQRDSOU2L4XA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.761563385 (2h38m42.542496775s) [noise=true , address=77.90.31.18:4822 , piece_id=YSR3AOJMALHQXCRAAS7HCGMYRBV2RENDMU3S4QN4NBWIIYBZ3XTQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.761921631 (2h38m42.542855017s) [noise=true , address=64.49.13.138:4488 , piece_id=WCVTP7RPSDFOAY7BTQCQRUNFUUHENQTHAXSAQW6KS5JBE73RIORA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.762057081 (2h38m42.542990472s) [noise=true , address=45.91.116.10:4093 , piece_id=YJWWVJFESPJ3D4EJ5WEREFCUGJPOIFTIFDDBB5UZELEEBMTA4GYA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.762305013 (2h38m42.543238402s) [noise=true , address=104.36.82.53:4128 , piece_id=SHXGTCA35HBFRIIRA26SNBUNG75PQ5AW4QP23V7OKXAFYA6BS6OQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.762653757 (2h38m42.543587145s) [noise=true , address=104.36.82.56:4229 , piece_id=KWMKQB6ZKAZX737Y5DHDUWHIQD3AJSHSP4MZKDX3T2WN6PK5ZQJA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.763559754 (2h38m42.54449312s) [noise=true , address=104.36.82.54:4159 , piece_id=QCG75UKHQ345V3OTOBAUVVE6ORLCLND5AS5L77TU7KVSL6TZZ2TA, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.766260892 (2h38m42.54719428s) [noise=true , address=104.36.82.19:4432 , piece_id=NZVRAYJXBDVW47VUGNPYP4B2XQ4NUB4I42555ZB35HAR6NT4FJFQ, seg_pos={0 0}, upload=2437]: Uploading piece... Jul 27 00:30:26.857899016 (2h38m42.638832381s) [upload=2437]: Next returned nil segment Jul 27 00:30:26.857916404 (2h38m42.638849772s) [upload=2437]: Waiting for error group managing segments... Jul 27 00:30:26.895736919 (2h38m42.676670282s) [noise=true , address=79.127.219.38:4003 , piece_id=HQMHFFMF5RTMY6745SMZMWONAUZHOC6BN22CFZ2TFV2PDQYPMULQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:26.901797700 (2h38m42.682731078s) [noise=true , address=77.90.31.16:4759 , piece_id=YBKDJVPNOEJS7OES46CDLEGMEITIZXFEI4M72ZYQX27J4Q3MTOSQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:26.941352483 (2h38m42.722285845s) [noise=true , address=77.90.31.21:4888 , piece_id=3EZ5Y5VMPSQJJDHKSBFYJPKPMTYNL2TIJMZLXE3KQNMN4IITFKIQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:26.970728884 (2h38m42.751662254s) [noise=true , address=77.90.31.18:4822 , piece_id=YSR3AOJMALHQXCRAAS7HCGMYRBV2RENDMU3S4QN4NBWIIYBZ3XTQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:27.193091405 (2h38m42.974024785s) [noise=true , address=45.91.116.10:4093 , piece_id=YJWWVJFESPJ3D4EJ5WEREFCUGJPOIFTIFDDBB5UZELEEBMTA4GYA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:27.704633744 (2h38m43.485567115s) [noise=true , address=79.127.220.97:4361 , piece_id=CFYELNCF25IV7UH7DZQ44FORQNPN3AUDYUWZTMI4Y2MKNQYZ3BJQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:27.992617758 (2h38m43.773551122s) [noise=true , address=64.49.13.138:4488 , piece_id=WCVTP7RPSDFOAY7BTQCQRUNFUUHENQTHAXSAQW6KS5JBE73RIORA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:28.216496748 (2h38m43.99743013s) [noise=true , address=156.146.43.225:4434 , piece_id=NR64SDAEX7AV2JSAPI53NHWJTCDS2Z4JUGBBNR7ZRLA4O56LMEGQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:29.463318792 (2h38m45.244252186s) [noise=true , address=77.90.31.13:4636 , piece_id=Q445H2SJ6ZEOI5GLWLPAWPYEYS6K4X5BP5HJBVSCDKK3ICEQF4BA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:30.282690166 (2h38m46.06362353s) [noise=true , address=77.90.31.19:4871 , piece_id=HGGMRVOTBRBKCKUW2KBIHVXJEQBVNKGT5XD7AMY4KQRDSOU2L4XA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:35.765953281 (2h38m51.546886648s) [noise=true , address=79.127.201.215:4349 , piece_id=MW2CVLRLZT3O3XETRD5XJXJX3QGURHTXRMSBST4MZ3KVVV34PCRQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:30:52.798238786 (2h39m8.579172172s) [noise=true , address=77.90.31.12:4617 , piece_id=G5M3FQUB6XYM5JST2XDUWJLUT7AEQ445AGIS3C7UIJVUNF6SW2UQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: <nil> Jul 27 00:31:26.681213804 (2h39m42.462147182s) [noise=true , address=104.36.82.53:4128 , piece_id=SHXGTCA35HBFRIIRA26SNBUNG75PQ5AW4QP23V7OKXAFYA6BS6OQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: rpc: dial tcp 104.36.82.53:4128: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:26.681303735 (2h39m42.4622371s) [noise=true , address=104.36.82.46:4482 , piece_id=2ZDWBSBPM5ZYALK2LLM7ZS52CAF6SELSMGRFFQRXVGEWKDPCRJPA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:26.681371926 (2h39m42.462305292s) [noise=true , address=104.36.82.47:4542 , piece_id=YLHHFQFRQ777FOACE2ZWJ4W555GKMJ75QO4IITBVLCHM77235KBA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:129FwaCA1FbzJbmRNzhQeVgygicJYHWkSS9fLnJbAG7KZPpokLU, address:104.36.82.47:4542): protocol: write tcp 159.223.143.39:60858->104.36.82.47:4542: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:26.681449275 (2h39m42.462382636s) [noise=true , address=104.36.82.19:4432 , piece_id=NZVRAYJXBDVW47VUGNPYP4B2XQ4NUB4I42555ZB35HAR6NT4FJFQ, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:12oh9XZaiLhaZdMyq1cx88yrrG9C9sHtH3RTA6W2Ly39t23ArSQ, address:104.36.82.19:4432): protocol: rpc: dial tcp 104.36.82.19:4432: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:26.681497320 (2h39m42.462430682s) [noise=true , address=104.36.82.56:4229 , piece_id=KWMKQB6ZKAZX737Y5DHDUWHIQD3AJSHSP4MZKDX3T2WN6PK5ZQJA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:104.36.82.56:4229): protocol: rpc: dial tcp 104.36.82.56:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:26.681691579 (2h39m42.46262496s) [noise=true , address=104.36.82.54:4159 , piece_id=QCG75UKHQ345V3OTOBAUVVE6ORLCLND5AS5L77TU7KVSL6TZZ2TA, seg_pos={0 0}, upload=2437]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:104.36.82.54:4159): protocol: write tcp 159.223.143.39:58284->104.36.82.54:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:31:26.681761093 (2h39m42.462694484s) [seg_pos={0 0}, upload=2437]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:31:26.708293966 (2h39m42.489227345s) [upload=2437]: Done uploading segments... Jul 27 00:31:26.708326568 (2h39m42.489259953s) [upload=2437]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.030003126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [XX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.X] duration: 30.033702022s FAIL (2025-07-27 00:31:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.033801056s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X..X......X.XX....X...X.................XX....X......X] duration: 30.001978905s FAIL (2025-07-27 00:31:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002011123s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..X.X........X...XX...............................X.........] duration: 44.227402898s FAIL (2025-07-27 00:30:57 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.227427659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 251.711027ms FAIL (2025-07-27 00:30:41 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 251.747449ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029713198s FAIL (2025-07-27 00:30:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== :4539 , piece_id=P62QYOR5HDVOXK66SON6ZAPNQ5UC6UQAFYK2OJ3Q4JWIDYUE746A, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.661934430 (2h37m42.442867818s) [noise=true , address=77.90.31.18:4822 , piece_id=4HZEIBK6BQT7U7I2W72FM5O35W7OXBQLL46G6BD3U7KOFD4KBX5Q, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.662314091 (2h37m42.443247474s) [noise=true , address=104.36.82.55:4225 , piece_id=43RTKCHDLS75BHHYKTNKIYCN5YXIORC2SLHSJFUOHVWR5FOJUIWA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.663392193 (2h37m42.444325584s) [noise=true , address=104.36.82.46:4500 , piece_id=QTT4DYSRCSLD2X432VTCGZVFNXXPF4SVCVLZV7UNMFUL6Q3TYUAQ, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.664032763 (2h37m42.444966149s) [noise=true , address=79.127.205.245:4228 , piece_id=NDBH5VMWRL7GYFNV2SYMS3Q3OT3TDDJNI4QPHVWUF53Y5JT5NQBQ, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.664450703 (2h37m42.445384143s) [noise=true , address=77.90.31.11:4559 , piece_id=O3MDXB3TP2E4RWDX74VNILW3PAKJBVV5OAE66V45C3KIJYYOTZQA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.664845441 (2h37m42.445778827s) [noise=true , address=79.127.205.234:4603 , piece_id=VNHLUYB532AQPXGY4C3DQ7JK4DRC7YW6RJZND7SDHI2SIPNHWIAA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.665253619 (2h37m42.446187005s) [noise=true , address=77.90.31.17:4780 , piece_id=MI6KK2MS7TT372RYYS2KJB2V4VVWAR6LEXBZVZ7FL5BNKSMMQESA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.665617326 (2h37m42.446550711s) [noise=true , address=79.127.219.38:4010 , piece_id=GVUZO4HFVRUCFZYHEB5JQJ4PG57KOJBW6LPWCYATBIYGDTMIJ5IA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.671138919 (2h37m42.452072308s) [noise=true , address=45.91.116.10:4093 , piece_id=2XHOKOU4JMUKRZV5QOFVZUSGKCMXHQ6DEGQBLKNK66J3EQRRTSVQ, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.671350640 (2h37m42.452284024s) [noise=true , address=104.36.82.53:4138 , piece_id=AJWKJRMD4ST4TD6B6GL4ZFAM3F6JR32LHHZOPQN53Y437P2PZG4Q, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.671782480 (2h37m42.45271586s) [noise=true , address=45.91.116.4:4964 , piece_id=N6T25Q3JMP4CXHCDEQZMDT5UBOJQYQBXTBEXRIOZ64SD33ZBBULA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.672989070 (2h37m42.453922454s) [noise=true , address=104.36.82.54:4167 , piece_id=P5YVEAEU527VJLXTHS2SYAEEFLZRD5AAAQSR3DRMSMSGEU4GTREA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.673375781 (2h37m42.454309166s) [noise=true , address=45.91.116.14:4206 , piece_id=R4SXHGOIPGCHEQL4ZPWJIPKNCQMIZCU4YCZZBVFGBEJALXCQFMEQ, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.674160299 (2h37m42.455093712s) [noise=true , address=79.127.201.215:4358 , piece_id=YA3RKYWGE46EWP3WGVBYOEGLHABX3NTWXOAZRIEARO3MZRV27L6Q, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.674679999 (2h37m42.455613387s) [noise=true , address=77.90.31.12:4600 , piece_id=GXAZ6PGFHS2VOMDPCNEHSA5AUJBZK6CVT5C2YCXPKXXAWJUJJD4Q, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.675145437 (2h37m42.456078823s) [noise=true , address=64.49.13.138:4483 , piece_id=OZXYLSNSMIPFEFEPJOVCZCLFZFPC5GAPZ4OMQ6GWBMJGNBH5XEUA, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.676664733 (2h37m42.457598121s) [noise=true , address=79.127.219.47:4048 , piece_id=GQ4IITCJE4IWMUMIMTVGTDNMDVQHIENU74YHDJNJJPVGYV4Q4JBQ, seg_pos={0 0}, upload=2421]: Uploading piece... Jul 27 00:29:26.782893377 (2h37m42.563826744s) [upload=2421]: Next returned nil segment Jul 27 00:29:26.782915026 (2h37m42.563848414s) [upload=2421]: Waiting for error group managing segments... Jul 27 00:29:26.816774288 (2h37m42.597707669s) [noise=true , address=79.127.219.38:4010 , piece_id=GVUZO4HFVRUCFZYHEB5JQJ4PG57KOJBW6LPWCYATBIYGDTMIJ5IA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:27.097229720 (2h37m42.878163102s) [noise=true , address=79.127.219.47:4048 , piece_id=GQ4IITCJE4IWMUMIMTVGTDNMDVQHIENU74YHDJNJJPVGYV4Q4JBQ, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:29:27.138066738 (2h37m42.91900011s) [noise=true , address=77.90.31.18:4822 , piece_id=4HZEIBK6BQT7U7I2W72FM5O35W7OXBQLL46G6BD3U7KOFD4KBX5Q, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:27.334775925 (2h37m43.115709293s) [noise=true , address=79.127.205.245:4228 , piece_id=NDBH5VMWRL7GYFNV2SYMS3Q3OT3TDDJNI4QPHVWUF53Y5JT5NQBQ, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:27.380866201 (2h37m43.161799569s) [noise=true , address=64.49.13.138:4483 , piece_id=OZXYLSNSMIPFEFEPJOVCZCLFZFPC5GAPZ4OMQ6GWBMJGNBH5XEUA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:27.382245435 (2h37m43.163178812s) [noise=true , address=79.127.205.234:4603 , piece_id=VNHLUYB532AQPXGY4C3DQ7JK4DRC7YW6RJZND7SDHI2SIPNHWIAA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:29.241791083 (2h37m45.022724466s) [noise=true , address=77.90.31.12:4600 , piece_id=GXAZ6PGFHS2VOMDPCNEHSA5AUJBZK6CVT5C2YCXPKXXAWJUJJD4Q, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:30.633493226 (2h37m46.41442664s) [noise=true , address=45.91.116.10:4093 , piece_id=2XHOKOU4JMUKRZV5QOFVZUSGKCMXHQ6DEGQBLKNK66J3EQRRTSVQ, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:32.319644319 (2h37m48.100577696s) [noise=true , address=45.91.116.4:4964 , piece_id=N6T25Q3JMP4CXHCDEQZMDT5UBOJQYQBXTBEXRIOZ64SD33ZBBULA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:29:44.407666309 (2h38m0.188599691s) [noise=true , address=45.91.116.14:4206 , piece_id=R4SXHGOIPGCHEQL4ZPWJIPKNCQMIZCU4YCZZBVFGBEJALXCQFMEQ, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:30:01.041240194 (2h38m16.82217357s) [noise=true , address=79.127.201.215:4358 , piece_id=YA3RKYWGE46EWP3WGVBYOEGLHABX3NTWXOAZRIEARO3MZRV27L6Q, seg_pos={0 0}, upload=2421]: Done uploading piece. err: <nil> Jul 27 00:30:26.583344427 (2h38m42.36427779s) [noise=true , address=104.36.82.54:4167 , piece_id=P5YVEAEU527VJLXTHS2SYAEEFLZRD5AAAQSR3DRMSMSGEU4GTREA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:104.36.82.54:4167): protocol: write tcp 159.223.143.39:43782->104.36.82.54:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.583504686 (2h38m42.364438069s) [noise=true , address=104.36.82.55:4225 , piece_id=43RTKCHDLS75BHHYKTNKIYCN5YXIORC2SLHSJFUOHVWR5FOJUIWA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: write tcp 159.223.143.39:45350->104.36.82.55:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.583663569 (2h38m42.364596957s) [noise=true , address=104.36.82.46:4500 , piece_id=QTT4DYSRCSLD2X432VTCGZVFNXXPF4SVCVLZV7UNMFUL6Q3TYUAQ, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 159.223.143.39:50636->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.583768554 (2h38m42.364701941s) [noise=true , address=77.90.31.17:4780 , piece_id=MI6KK2MS7TT372RYYS2KJB2V4VVWAR6LEXBZVZ7FL5BNKSMMQESA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:1XfvmVdBudtFnKwNxCxuGbQMS6HSSkVwPkYmffa5X6e6uYNcuF, address:77.90.31.17:4780): protocol: write tcp 159.223.143.39:54900->77.90.31.17:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.583975215 (2h38m42.364908592s) [noise=true , address=104.36.82.47:4539 , piece_id=P62QYOR5HDVOXK66SON6ZAPNQ5UC6UQAFYK2OJ3Q4JWIDYUE746A, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:126g75XgAyFYcgnaibseqoGiz7cK6z3n6gPMcu9TrdjuZUYEHnZ, address:104.36.82.47:4539): protocol: rpc: dial tcp 104.36.82.47:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.584453239 (2h38m42.365386618s) [noise=true , address=77.90.31.11:4559 , piece_id=O3MDXB3TP2E4RWDX74VNILW3PAKJBVV5OAE66V45C3KIJYYOTZQA, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 159.223.143.39:45700->77.90.31.11:4559: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.584473216 (2h38m42.365406583s) [noise=true , address=104.36.82.53:4138 , piece_id=AJWKJRMD4ST4TD6B6GL4ZFAM3F6JR32LHHZOPQN53Y437P2PZG4Q, seg_pos={0 0}, upload=2421]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: write tcp 159.223.143.39:34800->104.36.82.53:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:26.584563532 (2h38m42.36549691s) [seg_pos={0 0}, upload=2421]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:30:26.610545833 (2h38m42.391479205s) [upload=2421]: Done uploading segments... Jul 27 00:30:26.610576253 (2h38m42.391509619s) [upload=2421]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.029748812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..] duration: 1.849956926s FAIL (2025-07-27 00:30:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 159.223.143.39:51274->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511244882 (2h36m52.29217826s) [noise=true , address=77.90.31.16:4733 , piece_id=A7J2Y3623TGOT5MNNZB3FDESJ2AC56ECZB6HJNYMZ2XY6SFF7FRA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:36.511286441 (2h36m52.292219832s) [seg_pos={0 0}, upload=2401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:28:36.543236445 (2h36m52.324169827s) [upload=2401]: Done uploading segments... Jul 27 00:28:36.543304166 (2h36m52.324237549s) [upload=2401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:28:36.635198096 (2h36m52.416131462s) [upload=2408]: Starting upload Jul 27 00:28:36.635282791 (2h36m52.416216158s) [upload=2408]: Done starting upload Jul 27 00:28:36.635390123 (2h36m52.416323499s) [upload=2408]: Uploading segments... Jul 27 00:28:36.635558688 (2h36m52.416492068s) [upload=2408]: Got next segment. Inline: false Jul 27 00:28:36.635615791 (2h36m52.416549176s) [upload=2408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:28:36.635624207 (2h36m52.416557589s) [upload=2408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:28:36.671104709 (2h36m52.452038096s) [seg_pos={0 0}, upload=2408]: Begin upload segment... Jul 27 00:28:36.671182809 (2h36m52.452116177s) [seg_pos={0 0}, upload=2408]: Done begin upload segment. Jul 27 00:28:36.671242667 (2h36m52.45217605s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.671699881 (2h36m52.452633265s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672237090 (2h36m52.453170477s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672600157 (2h36m52.453533546s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672929956 (2h36m52.453863336s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673266991 (2h36m52.45420038s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673621446 (2h36m52.454554833s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673960690 (2h36m52.454894077s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676064765 (2h36m52.456998144s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676576481 (2h36m52.45750986s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676952754 (2h36m52.457886136s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677320377 (2h36m52.458253757s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677690801 (2h36m52.458624182s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678050323 (2h36m52.458983704s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678430398 (2h36m52.45936378s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678798637 (2h36m52.459732018s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.774901091 (2h36m52.555834474s) [upload=2408]: Next returned nil segment Jul 27 00:28:36.774921252 (2h36m52.555854639s) [upload=2408]: Waiting for error group managing segments... Jul 27 00:28:36.778853374 (2h36m52.559786751s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.778997842 (2h36m52.559931219s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779321513 (2h36m52.560254881s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779807528 (2h36m52.560740898s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779925920 (2h36m52.560859296s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780369990 (2h36m52.56130336s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780649742 (2h36m52.561583126s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780760157 (2h36m52.561693527s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781178115 (2h36m52.562111492s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781195263 (2h36m52.562128639s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781643788 (2h36m52.562577166s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:38.197748474 (2h36m53.978681856s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:29:06.600686278 (2h37m22.381619658s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:55150->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600775029 (2h37m22.381708408s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:57018->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600786915 (2h37m22.381720287s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:56824->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600852455 (2h37m22.381785822s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 159.223.143.39:57226->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600896768 (2h37m22.381830156s) [seg_pos={0 0}, upload=2408]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:29:06.629712908 (2h37m22.410646275s) [upload=2408]: Done uploading segments... Jul 27 00:29:06.629739569 (2h37m22.410672946s) [upload=2408]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.849993826s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...X.X........X...XX...............................X........] duration: 51.214764286s FAIL (2025-07-27 00:30:12 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.214961767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX.] duration: 2.013216973s FAIL (2025-07-27 00:29:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 159.223.143.39:51274->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511244882 (2h36m52.29217826s) [noise=true , address=77.90.31.16:4733 , piece_id=A7J2Y3623TGOT5MNNZB3FDESJ2AC56ECZB6HJNYMZ2XY6SFF7FRA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:36.511286441 (2h36m52.292219832s) [seg_pos={0 0}, upload=2401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:28:36.543236445 (2h36m52.324169827s) [upload=2401]: Done uploading segments... Jul 27 00:28:36.543304166 (2h36m52.324237549s) [upload=2401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:28:36.635198096 (2h36m52.416131462s) [upload=2408]: Starting upload Jul 27 00:28:36.635282791 (2h36m52.416216158s) [upload=2408]: Done starting upload Jul 27 00:28:36.635390123 (2h36m52.416323499s) [upload=2408]: Uploading segments... Jul 27 00:28:36.635558688 (2h36m52.416492068s) [upload=2408]: Got next segment. Inline: false Jul 27 00:28:36.635615791 (2h36m52.416549176s) [upload=2408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:28:36.635624207 (2h36m52.416557589s) [upload=2408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:28:36.671104709 (2h36m52.452038096s) [seg_pos={0 0}, upload=2408]: Begin upload segment... Jul 27 00:28:36.671182809 (2h36m52.452116177s) [seg_pos={0 0}, upload=2408]: Done begin upload segment. Jul 27 00:28:36.671242667 (2h36m52.45217605s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.671699881 (2h36m52.452633265s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672237090 (2h36m52.453170477s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672600157 (2h36m52.453533546s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672929956 (2h36m52.453863336s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673266991 (2h36m52.45420038s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673621446 (2h36m52.454554833s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673960690 (2h36m52.454894077s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676064765 (2h36m52.456998144s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676576481 (2h36m52.45750986s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676952754 (2h36m52.457886136s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677320377 (2h36m52.458253757s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677690801 (2h36m52.458624182s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678050323 (2h36m52.458983704s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678430398 (2h36m52.45936378s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678798637 (2h36m52.459732018s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.774901091 (2h36m52.555834474s) [upload=2408]: Next returned nil segment Jul 27 00:28:36.774921252 (2h36m52.555854639s) [upload=2408]: Waiting for error group managing segments... Jul 27 00:28:36.778853374 (2h36m52.559786751s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.778997842 (2h36m52.559931219s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779321513 (2h36m52.560254881s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779807528 (2h36m52.560740898s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779925920 (2h36m52.560859296s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780369990 (2h36m52.56130336s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780649742 (2h36m52.561583126s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780760157 (2h36m52.561693527s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781178115 (2h36m52.562111492s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781195263 (2h36m52.562128639s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781643788 (2h36m52.562577166s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:38.197748474 (2h36m53.978681856s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:29:06.600686278 (2h37m22.381619658s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:55150->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600775029 (2h37m22.381708408s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:57018->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600786915 (2h37m22.381720287s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:56824->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600852455 (2h37m22.381785822s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 159.223.143.39:57226->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600896768 (2h37m22.381830156s) [seg_pos={0 0}, upload=2408]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:29:06.629712908 (2h37m22.410646275s) [upload=2408]: Done uploading segments... Jul 27 00:29:06.629739569 (2h37m22.410672946s) [upload=2408]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.013243864s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X..X......X.XX....X...X.................XX....X......] duration: 1.819044687s FAIL (2025-07-27 00:29:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.819085583s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.] duration: 1.976758114s FAIL (2025-07-27 00:29:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.97680179s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX] duration: 30.042374892s FAIL (2025-07-27 00:29:34 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.04241149s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 231.015149ms FAIL (2025-07-27 00:29:32 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 231.076638ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.032699292s FAIL (2025-07-27 00:29:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== _pos={0 0}, upload=2406]: Begin upload segment... Jul 27 00:28:26.566373976 (2h36m42.347307348s) [seg_pos={0 0}, upload=2406]: Done begin upload segment. Jul 27 00:28:26.566413627 (2h36m42.347347013s) [noise=true , address=104.36.82.53:4134 , piece_id=JYTJJMDG5NCTTVC46XAMKLLKMZ5P3NIPSB7UWX4BY5W53BUFU4LA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.566936420 (2h36m42.347869808s) [noise=true , address=79.127.205.245:4225 , piece_id=V4OVOLWR6F4H2ILVCEIZA7SPDXDN6XR7AFD5FM3VCXYOLEOKTNKA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.567288935 (2h36m42.348222311s) [noise=true , address=77.90.31.19:4875 , piece_id=GRNGLQ67YDNHPU6BJCQJSCV2AC4HJOUT3LNG545FPZAMYZWGOW2A, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.567426154 (2h36m42.34835954s) [noise=true , address=104.36.82.52:4095 , piece_id=GMRVDC6YFYLISN34VAU5SZALSOBRUQLHUP32WSFHY732Z6XTAONQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.567973431 (2h36m42.348907014s) [noise=true , address=77.90.31.21:4897 , piece_id=LYFCOCMX7O37WMMFDBMYEP5XNBAD7G74DEBMWTX6ZSCVQS6GFS5Q, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.568027009 (2h36m42.348960396s) [noise=true , address=45.91.116.8:4062 , piece_id=6HXNRHI72DI7NEBRNDFSKXV4O7BWSMWMFR5T2CYZALMT63H3G2IA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.568126507 (2h36m42.349059899s) [noise=true , address=77.90.31.12:4621 , piece_id=6NGMIDF6AK56CTICEYNWNTEW5X7MY6UBONZBZLPX24WF7K4332YQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.568579578 (2h36m42.349512969s) [noise=true , address=104.36.82.56:4261 , piece_id=65XTR42O43FJU57DZH4Z6MTUUA2WLBV7KIYTEZ2PVC6ZM3T34BBQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.568615436 (2h36m42.349548824s) [noise=true , address=104.36.82.11:4138 , piece_id=U776CGTFPNHFUW6E7LQUCCOGWWZV3LO37EEMTTSPPKTAHLP4E7GQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.568998998 (2h36m42.349932436s) [noise=true , address=104.36.82.55:4220 , piece_id=YV4LB7XLT5OPFA7JUSDW4UPAWEN6P34TSKARHZHLPWFUYRFDLVIA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.569016887 (2h36m42.349950274s) [noise=true , address=64.49.13.140:4514 , piece_id=HRW6BF45DA5CB4WRWWE55G3CMS74CDPCIZD6QWHWMEDZKMT42MBQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.569440076 (2h36m42.350373464s) [noise=true , address=79.127.219.38:4004 , piece_id=VXO5WHWP3S4HRTKPAJFCRCFLWFZESFOKFEYPQPQDGSCMCOQWVAGA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.569911732 (2h36m42.350845118s) [noise=true , address=79.127.201.215:4360 , piece_id=LM5GL7CAU2XCGTVCJYXVTDBUHSXBJUPPIAZBSRTWDSBJG7N7MUEQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.571601249 (2h36m42.352534638s) [noise=true , address=64.49.13.138:4483 , piece_id=DRBL3BTL6WRQHUEDPHSSIUOVWMPXDJV4TZUKBFVZK66H4LAPNQYQ, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.572216434 (2h36m42.353149826s) [noise=true , address=104.36.82.47:4526 , piece_id=ADAI444AWHNQEHBEBAQTDJ25ZPHOVIJAUF736TLBYE2MKL7RIONA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.572709516 (2h36m42.353642904s) [noise=true , address=104.36.82.46:4484 , piece_id=TIUCHXHM5GFIA6VGZWXF6GVSPENIGNWKJSFGR5DT53URBVL5N6GA, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.573075818 (2h36m42.354009206s) [noise=true , address=79.127.201.214:4289 , piece_id=MLMZ5DQ3SUD2Z5VDDIAHT22YF5PKMKUYSB2ISV6BFXCX6WOGVX2Q, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.573597876 (2h36m42.354531256s) [noise=true , address=45.91.116.4:4964 , piece_id=5D76VB45ECVXTSHSZDWFHSRFE5GY4NS7D7ZSMLDSKGTQGQQDVF6Q, seg_pos={0 0}, upload=2406]: Uploading piece... Jul 27 00:28:26.623013151 (2h36m42.403946519s) [noise=true , address=77.90.31.19:4875 , piece_id=GRNGLQ67YDNHPU6BJCQJSCV2AC4HJOUT3LNG545FPZAMYZWGOW2A, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1xSFQaW3aVnAVDYE5r5ADmCGU95vWnshuZyte32W2YAj94avNn, address:77.90.31.19:4875): protocol: storage node overloaded, request limit: 2000 Jul 27 00:28:26.657378540 (2h36m42.438311908s) [upload=2406]: Next returned nil segment Jul 27 00:28:26.657396149 (2h36m42.438329526s) [upload=2406]: Waiting for error group managing segments... Jul 27 00:28:26.673084853 (2h36m42.454018235s) [noise=true , address=79.127.219.38:4004 , piece_id=VXO5WHWP3S4HRTKPAJFCRCFLWFZESFOKFEYPQPQDGSCMCOQWVAGA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:26.703091616 (2h36m42.484024987s) [noise=true , address=77.90.31.21:4897 , piece_id=LYFCOCMX7O37WMMFDBMYEP5XNBAD7G74DEBMWTX6ZSCVQS6GFS5Q, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:26.739229394 (2h36m42.520162772s) [noise=true , address=45.91.116.8:4062 , piece_id=6HXNRHI72DI7NEBRNDFSKXV4O7BWSMWMFR5T2CYZALMT63H3G2IA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:27.209815386 (2h36m42.990748762s) [noise=true , address=79.127.205.245:4225 , piece_id=V4OVOLWR6F4H2ILVCEIZA7SPDXDN6XR7AFD5FM3VCXYOLEOKTNKA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:27.323401171 (2h36m43.104334541s) [noise=true , address=64.49.13.140:4514 , piece_id=HRW6BF45DA5CB4WRWWE55G3CMS74CDPCIZD6QWHWMEDZKMT42MBQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:30.605373476 (2h36m46.386306854s) [noise=true , address=79.127.201.214:4289 , piece_id=MLMZ5DQ3SUD2Z5VDDIAHT22YF5PKMKUYSB2ISV6BFXCX6WOGVX2Q, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:35.281060700 (2h36m51.061994086s) [noise=true , address=45.91.116.4:4964 , piece_id=5D76VB45ECVXTSHSZDWFHSRFE5GY4NS7D7ZSMLDSKGTQGQQDVF6Q, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:36.214527059 (2h36m51.995460441s) [noise=true , address=64.49.13.138:4483 , piece_id=DRBL3BTL6WRQHUEDPHSSIUOVWMPXDJV4TZUKBFVZK66H4LAPNQYQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:57.034334653 (2h37m12.815268038s) [noise=true , address=77.90.31.12:4621 , piece_id=6NGMIDF6AK56CTICEYNWNTEW5X7MY6UBONZBZLPX24WF7K4332YQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:28:59.491551681 (2h37m15.272485045s) [noise=true , address=79.127.201.215:4360 , piece_id=LM5GL7CAU2XCGTVCJYXVTDBUHSXBJUPPIAZBSRTWDSBJG7N7MUEQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: <nil> Jul 27 00:29:26.493759888 (2h37m42.274693269s) [noise=true , address=104.36.82.55:4220 , piece_id=YV4LB7XLT5OPFA7JUSDW4UPAWEN6P34TSKARHZHLPWFUYRFDLVIA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: rpc: dial tcp 104.36.82.55:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:26.493884848 (2h37m42.274818216s) [noise=true , address=104.36.82.11:4138 , piece_id=U776CGTFPNHFUW6E7LQUCCOGWWZV3LO37EEMTTSPPKTAHLP4E7GQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: write tcp 159.223.143.39:53730->104.36.82.11:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:26.493956328 (2h37m42.274889706s) [noise=true , address=104.36.82.46:4484 , piece_id=TIUCHXHM5GFIA6VGZWXF6GVSPENIGNWKJSFGR5DT53URBVL5N6GA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: rpc: dial tcp 104.36.82.46:4484: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:26.494029799 (2h37m42.274963177s) [noise=true , address=104.36.82.52:4095 , piece_id=GMRVDC6YFYLISN34VAU5SZALSOBRUQLHUP32WSFHY732Z6XTAONQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1idvNc8SGRP7tkAckr2TLv7LnvftiRUTGcwy5GCrH5rVyDEK5N, address:104.36.82.52:4095): protocol: rpc: dial tcp 104.36.82.52:4095: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:26.494153785 (2h37m42.275087167s) [noise=true , address=104.36.82.56:4261 , piece_id=65XTR42O43FJU57DZH4Z6MTUUA2WLBV7KIYTEZ2PVC6ZM3T34BBQ, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: rpc: dial tcp 104.36.82.56:4261: i/o timeout; context deadline exceeded Jul 27 00:29:26.494318860 (2h37m42.275252239s) [noise=true , address=104.36.82.53:4134 , piece_id=JYTJJMDG5NCTTVC46XAMKLLKMZ5P3NIPSB7UWX4BY5W53BUFU4LA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: write tcp 159.223.143.39:55482->104.36.82.53:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:26.494396076 (2h37m42.275329455s) [noise=true , address=104.36.82.47:4526 , piece_id=ADAI444AWHNQEHBEBAQTDJ25ZPHOVIJAUF736TLBYE2MKL7RIONA, seg_pos={0 0}, upload=2406]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: rpc: dial tcp 104.36.82.47:4526: i/o timeout; conn closed: op Jul 27 00:29:26.494455547 (2h37m42.275388985s) [seg_pos={0 0}, upload=2406]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:29:26.524227752 (2h37m42.305161126s) [upload=2406]: Done uploading segments... Jul 27 00:29:26.524260252 (2h37m42.305193629s) [upload=2406]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.032724668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....X.X........X...XX...............................X.......] duration: 56.870061112s FAIL (2025-07-27 00:29:21 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.870078517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX] duration: 30.031167187s FAIL (2025-07-27 00:29:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 159.223.143.39:51274->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511244882 (2h36m52.29217826s) [noise=true , address=77.90.31.16:4733 , piece_id=A7J2Y3623TGOT5MNNZB3FDESJ2AC56ECZB6HJNYMZ2XY6SFF7FRA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:36.511286441 (2h36m52.292219832s) [seg_pos={0 0}, upload=2401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:28:36.543236445 (2h36m52.324169827s) [upload=2401]: Done uploading segments... Jul 27 00:28:36.543304166 (2h36m52.324237549s) [upload=2401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:28:36.635198096 (2h36m52.416131462s) [upload=2408]: Starting upload Jul 27 00:28:36.635282791 (2h36m52.416216158s) [upload=2408]: Done starting upload Jul 27 00:28:36.635390123 (2h36m52.416323499s) [upload=2408]: Uploading segments... Jul 27 00:28:36.635558688 (2h36m52.416492068s) [upload=2408]: Got next segment. Inline: false Jul 27 00:28:36.635615791 (2h36m52.416549176s) [upload=2408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:28:36.635624207 (2h36m52.416557589s) [upload=2408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:28:36.671104709 (2h36m52.452038096s) [seg_pos={0 0}, upload=2408]: Begin upload segment... Jul 27 00:28:36.671182809 (2h36m52.452116177s) [seg_pos={0 0}, upload=2408]: Done begin upload segment. Jul 27 00:28:36.671242667 (2h36m52.45217605s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.671699881 (2h36m52.452633265s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672237090 (2h36m52.453170477s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672600157 (2h36m52.453533546s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.672929956 (2h36m52.453863336s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673266991 (2h36m52.45420038s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673621446 (2h36m52.454554833s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.673960690 (2h36m52.454894077s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676064765 (2h36m52.456998144s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676576481 (2h36m52.45750986s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.676952754 (2h36m52.457886136s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677320377 (2h36m52.458253757s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.677690801 (2h36m52.458624182s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678050323 (2h36m52.458983704s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678430398 (2h36m52.45936378s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.678798637 (2h36m52.459732018s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Uploading piece... Jul 27 00:28:36.774901091 (2h36m52.555834474s) [upload=2408]: Next returned nil segment Jul 27 00:28:36.774921252 (2h36m52.555854639s) [upload=2408]: Waiting for error group managing segments... Jul 27 00:28:36.778853374 (2h36m52.559786751s) [noise=true , address=45.91.116.4:4940 , piece_id=SBOSHLKM6UYPJK4KA4OPXWI42UH2VW3RXPUYXIYGIYPMOH4KTIAQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.778997842 (2h36m52.559931219s) [noise=true , address=45.91.116.12:4226 , piece_id=WW7F5L4QW7SUG5YGTGU7Z5TZCVJLFTXN55OCIUYS2IPDDFLPA53Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779321513 (2h36m52.560254881s) [noise=true , address=77.90.31.21:4877 , piece_id=HG5T4FAGHANMO33MDR5WLJTHTNGQ53CFZHQOX36UX63ZJPRXCDOA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779807528 (2h36m52.560740898s) [noise=true , address=45.91.116.8:4047 , piece_id=UAZMUKSXBC6KY2VUDUS4PA3AUSM3TSVTSYBQJLBOAUIP5Y7NDVTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.779925920 (2h36m52.560859296s) [noise=true , address=45.91.116.13:4153 , piece_id=ICRMQTBBUN2GZJ3JSN4DD5DQRYWCSF4PPLAJA6BLX5R3DU76QPXA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780369990 (2h36m52.56130336s) [noise=true , address=77.90.31.14:4663 , piece_id=N5S4SSCW5CKARYJV3FLNUANJYAXLOI3IFHQ3HCHFBHOSI4Y7RYTA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780649742 (2h36m52.561583126s) [noise=true , address=45.91.116.7:4010 , piece_id=VODDQSW6M5YOHZ2APXXKLL7UMLOJGQWM7PLEABNKMBQQTBA4JDFQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.780760157 (2h36m52.561693527s) [noise=true , address=45.91.116.13:4154 , piece_id=2GKVATSZYNDBHROT3JDCVC734W2VJ55YIQWUBMH7RMGWPT6YOHNQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781178115 (2h36m52.562111492s) [noise=true , address=45.91.116.9:4261 , piece_id=5N6UJOYQSJWPXQ3JDK2SA557E7C5WYBFYKQLG3ULYR2GNUR46VAA, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781195263 (2h36m52.562128639s) [noise=true , address=45.91.116.11:4119 , piece_id=5YCGLZW2MRMHIZ25T3G3KBWIEE5ZRY76VRLWBN2VTRFRCQMEYWKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:36.781643788 (2h36m52.562577166s) [noise=true , address=45.91.116.7:4009 , piece_id=V7BCGLRYMWMK2EYBKCJ2VT3P7RMJOBPY6UG4EZR2APJWXHFTQVKQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:28:38.197748474 (2h36m53.978681856s) [noise=true , address=77.90.31.13:4628 , piece_id=7HNPAKF5GQNMRTFOZ4N5QWDX7CZNEJ2IJH7SQSHXDNXWBIOL6G5Q, seg_pos={0 0}, upload=2408]: Done uploading piece. err: <nil> Jul 27 00:29:06.600686278 (2h37m22.381619658s) [noise=true , address=104.36.82.13:4186 , piece_id=YBVRXJ6WUJVH44QQJWVY5ENPVDY3YE7JGUGRW5TRACJGGDIQ5T6A, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:55150->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600775029 (2h37m22.381708408s) [noise=true , address=104.36.82.5:4838 , piece_id=B4KO3OL3EDODK7KTBYHLD55PC7TEHHRKHPPX7ED2472XJBLEDVIQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:57018->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600786915 (2h37m22.381720287s) [noise=true , address=104.36.82.9:4006 , piece_id=5JGEG3Y6R5TBHJWCWGPHWNYXHO6ZOAHDFXKUSNW5LZZHOCCLURVQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:56824->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600852455 (2h37m22.381785822s) [noise=true , address=104.36.82.2:4732 , piece_id=PIP3MSNIUVDMGITFWV7MGA4R3R2FAUUP4KT6O3RRKWR5QPAF5VEQ, seg_pos={0 0}, upload=2408]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 159.223.143.39:57226->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:29:06.600896768 (2h37m22.381830156s) [seg_pos={0 0}, upload=2408]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:29:06.629712908 (2h37m22.410646275s) [upload=2408]: Done uploading segments... Jul 27 00:29:06.629739569 (2h37m22.410672946s) [upload=2408]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.03119018s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........X..X......X.XX....X...X.................XX....X.....] duration: 2.133487573s FAIL (2025-07-27 00:29:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.133516397s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 210.32042ms FAIL (2025-07-27 00:28:57 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 210.350119ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 214.890672ms FAIL (2025-07-27 00:28:40 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 214.918508ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...X] duration: 30.035028715s FAIL (2025-07-27 00:28:36 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 159.223.143.39:34290->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:50.445416404 (2h36m6.226349786s) [seg_pos={0 0}, upload=2396]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:27:50.445513078 (2h36m6.226446459s) [upload=2396]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:27:50.445518863 (2h36m6.226452227s) [upload=2396]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:27:50.503448342 (2h36m6.284381712s) [upload=2396]: Done uploading segments... Jul 27 00:27:50.503469984 (2h36m6.284403353s) [upload=2396]: Upload finished. err: <nil> Jul 27 00:28:06.550015467 (2h36m22.330948854s) [upload=2401]: Starting upload Jul 27 00:28:06.550183329 (2h36m22.33111669s) [upload=2401]: Done starting upload Jul 27 00:28:06.550579332 (2h36m22.33151271s) [upload=2401]: Uploading segments... Jul 27 00:28:06.550695736 (2h36m22.331629123s) [upload=2401]: Got next segment. Inline: false Jul 27 00:28:06.550739771 (2h36m22.331673154s) [upload=2401]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:28:06.550746924 (2h36m22.331680304s) [upload=2401]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:28:06.603796545 (2h36m22.384729931s) [seg_pos={0 0}, upload=2401]: Begin upload segment... Jul 27 00:28:06.603982778 (2h36m22.384916165s) [seg_pos={0 0}, upload=2401]: Done begin upload segment. Jul 27 00:28:06.604280686 (2h36m22.38521405s) [noise=true , address=77.90.31.19:4841 , piece_id=CCIVOOJRQLM2JO4GXFZIZGLAYZZZ2UGDXU6U3XNEI2BBIP5X4VAA, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.604341430 (2h36m22.385274816s) [noise=true , address=45.91.116.11:4117 , piece_id=XQMVHN7RFHG2O3R5I6CFYLKOMFHORZTOLHSREALUSDKS2FQNZMIQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.605853071 (2h36m22.386786453s) [noise=true , address=77.90.31.17:4770 , piece_id=GHZICB4PG7KZ27HWXUZR24OO6IQ3C3TZPZX4G3YVAVKF4G5KLZAA, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.606304326 (2h36m22.38723769s) [noise=true , address=104.36.82.5:4838 , piece_id=KVJM5EJUQSB6HY3DTNUX2CG23PSMSHEG7EABKV7BRUUVGUHJAESQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.607588272 (2h36m22.388521634s) [noise=true , address=77.90.31.12:4589 , piece_id=DXRH3VZGO2JX3S6YT5ICRJMDZLRWFXR2R6BZLN5ZL23FP23W2LIA, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.608585246 (2h36m22.389518609s) [noise=true , address=45.91.116.9:4263 , piece_id=4YVCO5KY5AT6UP3Z7BWMKACZQ65VUPYIYADJGB423YAX4LGOVOIQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.609659164 (2h36m22.390592542s) [noise=true , address=45.91.116.4:4937 , piece_id=ALBVLHGTQJDNQW5NX542OLZEMTAVC5JFXIYDFHNHO6VXILA4TSOQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.610711285 (2h36m22.391644656s) [noise=true , address=45.91.116.7:4009 , piece_id=CZFVW4PFVOA7MSMNS6IHLE47V4ZRXOLG3EE55ZGRCOJGUZGYIIMQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.612108187 (2h36m22.393041564s) [noise=true , address=104.36.82.16:4293 , piece_id=RRTUXELA7GTJYMOAPAYGZIB2DQYQ6G7LJSZI5KXCUMTPSC46FYSQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.615584609 (2h36m22.396517995s) [noise=true , address=77.90.31.14:4662 , piece_id=A4VMZD2WQZ6NCOTDOGRKWCHMMPVW3CVTIWM2R37V65XRZGMNRLLQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.618899846 (2h36m22.399833216s) [noise=true , address=104.36.82.14:4223 , piece_id=RB5FQ7CLN4QHNJWJ42Q3LHYGIRI5WIIIVVYTK4Q6TRX3TSIPSWJQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.622662503 (2h36m22.403595876s) [noise=true , address=45.91.116.4:4938 , piece_id=VOMPXWYLN2JO6XRMRJ7HZOLET6TUIUK2LHMIL7P3VBURFMDJ6MTQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.625082686 (2h36m22.406016057s) [noise=true , address=45.91.116.13:4153 , piece_id=6U7HI7CUE4H6JA5M2DNDOWVPM2DZSL36NFR64VN4QZG2JXKZC3RQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.627696691 (2h36m22.408630057s) [noise=true , address=45.91.116.12:4226 , piece_id=VDFJBBFYLQMMEIWNPPXLJBXATKRXW6W47G6DBQJDXMISZJFK33YQ, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.628254642 (2h36m22.409188019s) [noise=true , address=77.90.31.16:4733 , piece_id=A7J2Y3623TGOT5MNNZB3FDESJ2AC56ECZB6HJNYMZ2XY6SFF7FRA, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.630529342 (2h36m22.41146271s) [noise=true , address=45.91.116.6:4975 , piece_id=O53JBF4B6CS2BXIKETZPN6ELYFOIEJOGXJQSIO3GVWYZJIYNW4JA, seg_pos={0 0}, upload=2401]: Uploading piece... Jul 27 00:28:06.733959938 (2h36m22.514893319s) [upload=2401]: Next returned nil segment Jul 27 00:28:06.734004093 (2h36m22.514937478s) [upload=2401]: Waiting for error group managing segments... Jul 27 00:28:06.737109419 (2h36m22.518042799s) [noise=true , address=45.91.116.9:4263 , piece_id=4YVCO5KY5AT6UP3Z7BWMKACZQ65VUPYIYADJGB423YAX4LGOVOIQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.737154929 (2h36m22.518088299s) [noise=true , address=45.91.116.11:4117 , piece_id=XQMVHN7RFHG2O3R5I6CFYLKOMFHORZTOLHSREALUSDKS2FQNZMIQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.738142863 (2h36m22.519076247s) [noise=true , address=45.91.116.4:4937 , piece_id=ALBVLHGTQJDNQW5NX542OLZEMTAVC5JFXIYDFHNHO6VXILA4TSOQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.739076853 (2h36m22.520010231s) [noise=true , address=77.90.31.14:4662 , piece_id=A4VMZD2WQZ6NCOTDOGRKWCHMMPVW3CVTIWM2R37V65XRZGMNRLLQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.739552455 (2h36m22.520485819s) [noise=true , address=45.91.116.6:4975 , piece_id=O53JBF4B6CS2BXIKETZPN6ELYFOIEJOGXJQSIO3GVWYZJIYNW4JA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.739591458 (2h36m22.520524821s) [noise=true , address=45.91.116.7:4009 , piece_id=CZFVW4PFVOA7MSMNS6IHLE47V4ZRXOLG3EE55ZGRCOJGUZGYIIMQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.739641896 (2h36m22.52057526s) [noise=true , address=45.91.116.13:4153 , piece_id=6U7HI7CUE4H6JA5M2DNDOWVPM2DZSL36NFR64VN4QZG2JXKZC3RQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.740085966 (2h36m22.521019345s) [noise=true , address=45.91.116.4:4938 , piece_id=VOMPXWYLN2JO6XRMRJ7HZOLET6TUIUK2LHMIL7P3VBURFMDJ6MTQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:06.740848032 (2h36m22.521781409s) [noise=true , address=45.91.116.12:4226 , piece_id=VDFJBBFYLQMMEIWNPPXLJBXATKRXW6W47G6DBQJDXMISZJFK33YQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:08.153142773 (2h36m23.934076142s) [noise=true , address=77.90.31.19:4841 , piece_id=CCIVOOJRQLM2JO4GXFZIZGLAYZZZ2UGDXU6U3XNEI2BBIP5X4VAA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:08.885905413 (2h36m24.666838786s) [noise=true , address=77.90.31.12:4589 , piece_id=DXRH3VZGO2JX3S6YT5ICRJMDZLRWFXR2R6BZLN5ZL23FP23W2LIA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: <nil> Jul 27 00:28:36.510937108 (2h36m52.291870476s) [noise=true , address=77.90.31.17:4770 , piece_id=GHZICB4PG7KZ27HWXUZR24OO6IQ3C3TZPZX4G3YVAVKF4G5KLZAA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.510982410 (2h36m52.291915777s) [noise=true , address=104.36.82.14:4223 , piece_id=RB5FQ7CLN4QHNJWJ42Q3LHYGIRI5WIIIVVYTK4Q6TRX3TSIPSWJQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511012383 (2h36m52.291945755s) [noise=true , address=104.36.82.16:4293 , piece_id=RRTUXELA7GTJYMOAPAYGZIB2DQYQ6G7LJSZI5KXCUMTPSC46FYSQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511054847 (2h36m52.291988216s) [noise=true , address=104.36.82.5:4838 , piece_id=KVJM5EJUQSB6HY3DTNUX2CG23PSMSHEG7EABKV7BRUUVGUHJAESQ, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:51274->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:36.511244882 (2h36m52.29217826s) [noise=true , address=77.90.31.16:4733 , piece_id=A7J2Y3623TGOT5MNNZB3FDESJ2AC56ECZB6HJNYMZ2XY6SFF7FRA, seg_pos={0 0}, upload=2401]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:36.511286441 (2h36m52.292219832s) [seg_pos={0 0}, upload=2401]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:28:36.543236445 (2h36m52.324169827s) [upload=2401]: Done uploading segments... Jul 27 00:28:36.543304166 (2h36m52.324237549s) [upload=2401]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.035049769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXX] duration: 30.028810758s FAIL (2025-07-27 00:28:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.02883982s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.039830012s FAIL (2025-07-27 00:28:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== d=7BMKAKFYJSGBTDB64PQSNQLQCUDX64OOJJGAF3RBAYAERIZ3VNXA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.471231100 (2h35m42.252164508s) [noise=true , address=77.90.31.17:4792 , piece_id=I2EFY5REBRIIWNQYGBT5PETJ4PVPUQPQFL27TG23PHOTNI7L4T4Q, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.471356473 (2h35m42.25228986s) [noise=true , address=104.36.82.54:4163 , piece_id=22FA4PAWJB4FQJ43NSJRN3K6UA6PCDK7CKFHPGGH6TDOFAF7YOXQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.471772528 (2h35m42.252705915s) [noise=true , address=64.49.13.139:4478 , piece_id=7OUHO2VKK4ZIDDQSBPDEG7KEBQP7GO6J2HNTF7N5Z3O22TTRUR3Q, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.471962598 (2h35m42.252895985s) [noise=true , address=104.36.82.52:4097 , piece_id=43MACCM5RLSY3IXHPFFYNAKJIQY45E6Z24N7TISGAEQSKHZPEVJA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.472268991 (2h35m42.253202376s) [noise=true , address=79.127.205.245:4248 , piece_id=S25F3LV5UVV6UQRB27OSB76OAFVJJCQDC264D5SKIVFIH6B57KEQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.472509494 (2h35m42.253442871s) [noise=true , address=79.127.220.101:4041 , piece_id=MTYQKW3CLES2OMI5WQ3EBYUANNU7T73MPHR57FB7JNH6EBRGRODA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.473024889 (2h35m42.253958278s) [noise=true , address=79.127.205.246:4278 , piece_id=JVCYSTU6YKUOPIWYBA2NTVNAI4TTT6N6JMLVCVTR6MM22TDZ5NTA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.473104285 (2h35m42.254037675s) [noise=true , address=79.127.219.38:4006 , piece_id=FMHSWNMCOV5IM2XHDKQGULEJQXBGO6QXAW2GS6L6Y7JJ2TZ653DQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.473542915 (2h35m42.254476305s) [noise=true , address=104.36.82.46:4507 , piece_id=SHIBRAZNU65Q5EXKZNJZZ6YT4OJPE2OZOQJH6MUD3EHBJEX4LACA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.473570465 (2h35m42.254503849s) [noise=true , address=156.146.43.225:4461 , piece_id=B5K343ACIE27RPLTJIVTVE3DWVMRWJEOHWFHBNXKN7A4J3RS2L7Q, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.473959622 (2h35m42.25489301s) [noise=true , address=104.36.82.55:4202 , piece_id=NKVHLWTADQ6ALYP2Z5KBTRGY4YSC272YQXLOUTVWDRUFEYUCSKDA, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.474026753 (2h35m42.254960137s) [noise=true , address=104.36.82.47:4545 , piece_id=5HQAGGCDEHVM52TSNSX3O35ELQYNUO7DKHSJLWITZK4Q3KT7ZB6Q, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.474445495 (2h35m42.255378879s) [noise=true , address=104.36.82.56:4244 , piece_id=U2EIQHP5VR262NR53LOGXVQ7FOAJ5LYWDWFMB5YJRVIIXLXIKNMQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.474862688 (2h35m42.25579607s) [noise=true , address=77.90.31.13:4658 , piece_id=SKR62SSGFEGH43T2UECLP2HFAYRBNMNKTFMI53OKXB63JGGHJ74A, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.474873122 (2h35m42.255807289s) [noise=true , address=79.127.205.233:4642 , piece_id=WGKZBMVCFZH5SCCYCVDZYMBGLM2LW33QK45VYDCUXIYWMZUKVVHQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.478975067 (2h35m42.259908436s) [noise=true , address=79.127.220.97:4361 , piece_id=TXQYNILTXIKK4NAJ65EQ7JB45SS7PKYP7M5OZ5U2RYPVAWZ4476Q, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.480027718 (2h35m42.260961097s) [noise=true , address=104.36.82.11:4143 , piece_id=VWDZRME2V7OYFIK5EM7FPNHE2RQ4ALAZFEWOOWNTUZ5AGNFAPQMQ, seg_pos={0 0}, upload=2392]: Uploading piece... Jul 27 00:27:26.520474031 (2h35m42.301407417s) [noise=true , address=77.90.31.13:4658 , piece_id=SKR62SSGFEGH43T2UECLP2HFAYRBNMNKTFMI53OKXB63JGGHJ74A, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: storage node overloaded, request limit: 2000 Jul 27 00:27:26.554999884 (2h35m42.335933262s) [upload=2392]: Next returned nil segment Jul 27 00:27:26.555095212 (2h35m42.336028598s) [upload=2392]: Waiting for error group managing segments... Jul 27 00:27:26.559224449 (2h35m42.340157816s) [noise=true , address=79.127.219.38:4006 , piece_id=FMHSWNMCOV5IM2XHDKQGULEJQXBGO6QXAW2GS6L6Y7JJ2TZ653DQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:26.575349290 (2h35m42.356282673s) [noise=true , address=79.127.220.101:4041 , piece_id=MTYQKW3CLES2OMI5WQ3EBYUANNU7T73MPHR57FB7JNH6EBRGRODA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 27 00:27:26.626795101 (2h35m42.407728471s) [noise=true , address=156.146.43.225:4461 , piece_id=B5K343ACIE27RPLTJIVTVE3DWVMRWJEOHWFHBNXKN7A4J3RS2L7Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:1D4k7wnNtivVkHko5CSBuZH6zqMuFyfFfshkZwEPqnmpLads49, address:156.146.43.225:4461): protocol: storage node overloaded, request limit: 2000 Jul 27 00:27:27.001628703 (2h35m42.782562102s) [noise=true , address=79.127.220.97:4361 , piece_id=TXQYNILTXIKK4NAJ65EQ7JB45SS7PKYP7M5OZ5U2RYPVAWZ4476Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:27.061928944 (2h35m42.842862315s) [noise=true , address=79.127.205.235:4679 , piece_id=7BMKAKFYJSGBTDB64PQSNQLQCUDX64OOJJGAF3RBAYAERIZ3VNXA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:27.096086113 (2h35m42.877019476s) [noise=true , address=79.127.205.233:4642 , piece_id=WGKZBMVCFZH5SCCYCVDZYMBGLM2LW33QK45VYDCUXIYWMZUKVVHQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:27.703404942 (2h35m43.484338304s) [noise=true , address=79.127.205.245:4248 , piece_id=S25F3LV5UVV6UQRB27OSB76OAFVJJCQDC264D5SKIVFIH6B57KEQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:29.802564754 (2h35m45.583498125s) [noise=true , address=79.127.205.246:4278 , piece_id=JVCYSTU6YKUOPIWYBA2NTVNAI4TTT6N6JMLVCVTR6MM22TDZ5NTA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:27:33.581878489 (2h35m49.362811854s) [noise=true , address=64.49.13.139:4478 , piece_id=7OUHO2VKK4ZIDDQSBPDEG7KEBQP7GO6J2HNTF7N5Z3O22TTRUR3Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil> Jul 27 00:28:26.387999475 (2h36m42.168932852s) [noise=true , address=104.36.82.56:4244 , piece_id=U2EIQHP5VR262NR53LOGXVQ7FOAJ5LYWDWFMB5YJRVIIXLXIKNMQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12S1wh3PFQLPosjWmPs5iLfm4S5J6abXvoCfkhYz3KdvXCTaPRu, address:104.36.82.56:4244): protocol: write tcp 159.223.143.39:34230->104.36.82.56:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:26.388167306 (2h36m42.169100672s) [noise=true , address=77.90.31.17:4792 , piece_id=I2EFY5REBRIIWNQYGBT5PETJ4PVPUQPQFL27TG23PHOTNI7L4T4Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, address:77.90.31.17:4792): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.388340549 (2h36m42.16927393s) [noise=true , address=104.36.82.47:4545 , piece_id=5HQAGGCDEHVM52TSNSX3O35ELQYNUO7DKHSJLWITZK4Q3KT7ZB6Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.388591403 (2h36m42.169524782s) [noise=true , address=104.36.82.54:4163 , piece_id=22FA4PAWJB4FQJ43NSJRN3K6UA6PCDK7CKFHPGGH6TDOFAF7YOXQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: rpc: dial tcp 104.36.82.54:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:28:26.388706433 (2h36m42.169639802s) [noise=true , address=104.36.82.52:4097 , piece_id=43MACCM5RLSY3IXHPFFYNAKJIQY45E6Z24N7TISGAEQSKHZPEVJA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM, address:104.36.82.52:4097): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.388810431 (2h36m42.169743798s) [noise=true , address=104.36.82.55:4202 , piece_id=NKVHLWTADQ6ALYP2Z5KBTRGY4YSC272YQXLOUTVWDRUFEYUCSKDA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.389041442 (2h36m42.169974821s) [noise=true , address=104.36.82.46:4507 , piece_id=SHIBRAZNU65Q5EXKZNJZZ6YT4OJPE2OZOQJH6MUD3EHBJEX4LACA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.389138404 (2h36m42.17007178s) [noise=true , address=104.36.82.11:4143 , piece_id=VWDZRME2V7OYFIK5EM7FPNHE2RQ4ALAZFEWOOWNTUZ5AGNFAPQMQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12ipThd6PwZUpCXy3ML4qqMd9dmRLmxKjVNCNoT5wCg1szdWp8U, address:104.36.82.11:4143): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:28:26.389177161 (2h36m42.17011055s) [seg_pos={0 0}, upload=2392]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:28:26.424005881 (2h36m42.204939263s) [upload=2392]: Done uploading segments... Jul 27 00:28:26.424046045 (2h36m42.204979424s) [upload=2392]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.039877826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....X.X........X...XX...............................X......] duration: 42.550820116s FAIL (2025-07-27 00:28:24 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.550835699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........X..X......X.XX....X...X.................XX....X....] duration: 2.315862232s FAIL (2025-07-27 00:28:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.315890827s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........X..X......X.XX....X...X.................XX....X...] duration: 29.763447171s FAIL (2025-07-27 00:28:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.763469865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...] duration: 529.958679ms FAIL (2025-07-27 00:27:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 5.226875599 (2h33m1.007808984s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:35016->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 27 00:24:45.226924742 (2h33m1.007858123s) [seg_pos={0 0}, upload=2341]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:24:45.260403710 (2h33m1.041337086s) [upload=2341]: Done uploading segments... Jul 27 00:24:45.260436570 (2h33m1.041369954s) [upload=2341]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:25:13.130330845 (2h33m28.911264233s) [upload=2358]: Starting upload Jul 27 00:25:13.130427709 (2h33m28.911361097s) [upload=2358]: Done starting upload Jul 27 00:25:13.130504194 (2h33m28.911437572s) [upload=2358]: Uploading segments... Jul 27 00:25:13.130768741 (2h33m28.911702127s) [upload=2358]: Got next segment. Inline: false Jul 27 00:25:13.130825048 (2h33m28.911758432s) [upload=2358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:25:13.130832706 (2h33m28.911766093s) [upload=2358]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:25:13.156703675 (2h33m28.937637054s) [seg_pos={0 0}, upload=2358]: Begin upload segment... Jul 27 00:25:13.156790074 (2h33m28.937723451s) [seg_pos={0 0}, upload=2358]: Done begin upload segment. Jul 27 00:25:13.156839061 (2h33m28.937772446s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.156878026 (2h33m28.937811413s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157292455 (2h33m28.938225837s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157494201 (2h33m28.938427581s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157651501 (2h33m28.938584887s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158080485 (2h33m28.939013869s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158127600 (2h33m28.939060987s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158413346 (2h33m28.939346724s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158633735 (2h33m28.939567118s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158790022 (2h33m28.939723394s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.159238365 (2h33m28.940171753s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.160717151 (2h33m28.941650518s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.161474674 (2h33m28.942408056s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.162300982 (2h33m28.943234369s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.163746259 (2h33m28.94467965s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.164391211 (2h33m28.945324597s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.286660997 (2h33m29.067594379s) [upload=2358]: Next returned nil segment Jul 27 00:25:13.286684184 (2h33m29.067617547s) [upload=2358]: Waiting for error group managing segments... Jul 27 00:25:13.292602587 (2h33m29.073535969s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.292748204 (2h33m29.073681574s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.293326557 (2h33m29.074259939s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.293361832 (2h33m29.074295202s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.294304357 (2h33m29.075237733s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.295589750 (2h33m29.076523117s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.295634225 (2h33m29.076567588s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.296581902 (2h33m29.077515282s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.297185145 (2h33m29.078118508s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.325234980 (2h33m29.106168359s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.360841866 (2h33m29.141775245s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:43.100924665 (2h33m58.88185803s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 159.223.143.39:56852->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101183051 (2h33m58.882116415s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:25:43.101557492 (2h33m58.882490858s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101690637 (2h33m58.882624007s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:25:43.101784731 (2h33m58.882718095s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:54568->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101803464 (2h33m58.882736842s) [seg_pos={0 0}, upload=2358]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:25:43.126349574 (2h33m58.90728296s) [upload=2358]: Done uploading segments... Jul 27 00:25:43.126382341 (2h33m58.907315726s) [upload=2358]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 530.021974ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 205.808024ms FAIL (2025-07-27 00:27:43 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 205.833785ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......X.X........X...XX...............................X.....] duration: 59.192108616s FAIL (2025-07-27 00:27:42 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 59.192127209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXX] duration: 30.025111521s FAIL (2025-07-27 00:27:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.025131047s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX..] duration: 2.56986838s FAIL (2025-07-27 00:27:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 5.226875599 (2h33m1.007808984s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:35016->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 27 00:24:45.226924742 (2h33m1.007858123s) [seg_pos={0 0}, upload=2341]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:24:45.260403710 (2h33m1.041337086s) [upload=2341]: Done uploading segments... Jul 27 00:24:45.260436570 (2h33m1.041369954s) [upload=2341]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:25:13.130330845 (2h33m28.911264233s) [upload=2358]: Starting upload Jul 27 00:25:13.130427709 (2h33m28.911361097s) [upload=2358]: Done starting upload Jul 27 00:25:13.130504194 (2h33m28.911437572s) [upload=2358]: Uploading segments... Jul 27 00:25:13.130768741 (2h33m28.911702127s) [upload=2358]: Got next segment. Inline: false Jul 27 00:25:13.130825048 (2h33m28.911758432s) [upload=2358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:25:13.130832706 (2h33m28.911766093s) [upload=2358]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:25:13.156703675 (2h33m28.937637054s) [seg_pos={0 0}, upload=2358]: Begin upload segment... Jul 27 00:25:13.156790074 (2h33m28.937723451s) [seg_pos={0 0}, upload=2358]: Done begin upload segment. Jul 27 00:25:13.156839061 (2h33m28.937772446s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.156878026 (2h33m28.937811413s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157292455 (2h33m28.938225837s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157494201 (2h33m28.938427581s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157651501 (2h33m28.938584887s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158080485 (2h33m28.939013869s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158127600 (2h33m28.939060987s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158413346 (2h33m28.939346724s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158633735 (2h33m28.939567118s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158790022 (2h33m28.939723394s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.159238365 (2h33m28.940171753s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.160717151 (2h33m28.941650518s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.161474674 (2h33m28.942408056s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.162300982 (2h33m28.943234369s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.163746259 (2h33m28.94467965s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.164391211 (2h33m28.945324597s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.286660997 (2h33m29.067594379s) [upload=2358]: Next returned nil segment Jul 27 00:25:13.286684184 (2h33m29.067617547s) [upload=2358]: Waiting for error group managing segments... Jul 27 00:25:13.292602587 (2h33m29.073535969s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.292748204 (2h33m29.073681574s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.293326557 (2h33m29.074259939s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.293361832 (2h33m29.074295202s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.294304357 (2h33m29.075237733s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.295589750 (2h33m29.076523117s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.295634225 (2h33m29.076567588s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.296581902 (2h33m29.077515282s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.297185145 (2h33m29.078118508s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.325234980 (2h33m29.106168359s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.360841866 (2h33m29.141775245s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:43.100924665 (2h33m58.88185803s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 159.223.143.39:56852->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101183051 (2h33m58.882116415s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:25:43.101557492 (2h33m58.882490858s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101690637 (2h33m58.882624007s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:25:43.101784731 (2h33m58.882718095s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:54568->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101803464 (2h33m58.882736842s) [seg_pos={0 0}, upload=2358]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:25:43.126349574 (2h33m58.90728296s) [upload=2358]: Done uploading segments... Jul 27 00:25:43.126382341 (2h33m58.907315726s) [upload=2358]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.569917289s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........X..X......X.XX....X...X.................XX....X..] duration: 2.11329554s FAIL (2025-07-27 00:27:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.113316557s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027434565s FAIL (2025-07-27 00:27:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 414764 (2h34m42.091348144s) [upload=2377]: Got next segment. Inline: false Jul 27 00:26:26.310484559 (2h34m42.091417943s) [upload=2377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:26:26.310490187 (2h34m42.091423566s) [upload=2377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:26:26.343442614 (2h34m42.124375995s) [seg_pos={0 0}, upload=2377]: Begin upload segment... Jul 27 00:26:26.343522729 (2h34m42.124456115s) [seg_pos={0 0}, upload=2377]: Done begin upload segment. Jul 27 00:26:26.343575433 (2h34m42.124508821s) [noise=true , address=77.90.31.19:4875 , piece_id=D7IVBHHMQJKTMRRZWDUOQ6F3DQJ77TMOUC6ZKJOUBRBTRMFWRVGQ, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.344142525 (2h34m42.12507592s) [noise=true , address=77.90.31.16:4746 , piece_id=UDRB4TAYY5Z4PO7ZQFCPS5S7VX3LRQPU4Q5MPFB5C2F5LDI5JWXQ, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.344318450 (2h34m42.125251832s) [noise=true , address=79.127.205.246:4262 , piece_id=VZNLX55JACROS54O7JQWGBVQCYLRRS2NIBUCTVLL37GGENXO7CHQ, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.344735230 (2h34m42.125668638s) [noise=true , address=79.127.219.38:4018 , piece_id=QSFCC7YOPK2LOVYHTLFQ4MQT5AYP7XKOHO57XN5FEBDI557RIGKA, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.344869765 (2h34m42.125803388s) [noise=true , address=79.127.205.235:4691 , piece_id=ZHBI2JSZWGQE7Z2DTR6KXCBGCHTCHAUCZSKVVM75D3ZWJEWMTX3Q, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.345242507 (2h34m42.126175899s) [noise=true , address=64.49.13.140:4508 , piece_id=QFVYOC5QDA5S6STIGOW3LEHJ4QDUB4NOPLBOWNEHYXPAXJYALU5A, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.345448259 (2h34m42.126381635s) [noise=true , address=77.90.31.17:4793 , piece_id=YPY6GBN42ZLEA5GNKVT774ZY5WZTHLQZHPUJNBLHTCJK2FNHMG4A, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.345637254 (2h34m42.126570636s) [noise=true , address=77.90.31.18:4832 , piece_id=RDI5Y2IFQV5N7E454J6RC7IRA3UH5BUJLOTUH4KW22PMQ5N6FKAQ, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.345949417 (2h34m42.126882794s) [noise=true , address=64.49.13.137:4502 , piece_id=EKYOVB3VGNZEI33TUBBOEETIYNWOXX77HBKRAUQDC3TWU3CPKDYA, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.346135049 (2h34m42.127068445s) [noise=true , address=79.127.205.234:4608 , piece_id=W5HB4IJEJKCVH4H2TSDGOUSQMELEYJI4JNSBJG2C6H65K36OQOXA, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.346614775 (2h34m42.127548155s) [noise=true , address=104.36.82.46:4484 , piece_id=REF4XO4UYKOELCCQNRHPRZEBWDYSEZMNU2NMJFLWIDV3LRP6YNKA, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.347686402 (2h34m42.128619788s) [noise=true , address=104.36.82.52:4109 , piece_id=25MGIJGD3RJMLRNGKIEDS763YLSPR2FF5O5CSXWG2SE6AJL4UYIA, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.348021533 (2h34m42.128954919s) [noise=true , address=79.127.201.215:4360 , piece_id=45AIUENJKBJKZPLJOEMNO2OJVJYE4QHGQDGISWYU63PQSL3HJY2A, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.348172428 (2h34m42.129105817s) [noise=true , address=79.127.205.233:4656 , piece_id=CDINO2ITVWDVXNJ54UYBZME53OJB5R55YA2TYMA7Q5OU77KPUY5Q, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.348400972 (2h34m42.129334349s) [noise=true , address=77.90.31.21:4912 , piece_id=MVSDR6CZBQF2T7NEHXZJ23R6NZ2XOBX4T7WZZP76V5E4V2DZ5AAQ, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.348777975 (2h34m42.129711359s) [noise=true , address=104.36.82.47:4525 , piece_id=FQALJT36Z7N3V76SRSJWS3WZ3ZIP4736X76SJHHYMMWT2AULLG2A, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.348785465 (2h34m42.129718847s) [noise=true , address=45.91.116.9:4277 , piece_id=OHYZ6YEBAXIG4QQ5NPDAMFSMVMUG2AZUQ4HTM5XZVSHDKKKB5OZQ, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.349167556 (2h34m42.130100946s) [noise=true , address=79.127.220.101:4041 , piece_id=DJ4OC2BOSDKKYSHTDV7CPHPYEATJCFYBRNK5O47TRXI2SWOA3YYQ, seg_pos={0 0}, upload=2377]: Uploading piece... Jul 27 00:26:26.439333868 (2h34m42.220267235s) [upload=2377]: Next returned nil segment Jul 27 00:26:26.439357443 (2h34m42.220290827s) [upload=2377]: Waiting for error group managing segments... Jul 27 00:26:26.447634820 (2h34m42.228568189s) [noise=true , address=79.127.219.38:4018 , piece_id=QSFCC7YOPK2LOVYHTLFQ4MQT5AYP7XKOHO57XN5FEBDI557RIGKA, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil> Jul 27 00:26:26.500993895 (2h34m42.281927258s) [noise=true , address=77.90.31.18:4832 , piece_id=RDI5Y2IFQV5N7E454J6RC7IRA3UH5BUJLOTUH4KW22PMQ5N6FKAQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil> Jul 27 00:26:26.560868998 (2h34m42.341802397s) [noise=true , address=77.90.31.21:4912 , piece_id=MVSDR6CZBQF2T7NEHXZJ23R6NZ2XOBX4T7WZZP76V5E4V2DZ5AAQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil> Jul 27 00:26:26.974790596 (2h34m42.755723974s) [noise=true , address=79.127.205.235:4691 , piece_id=ZHBI2JSZWGQE7Z2DTR6KXCBGCHTCHAUCZSKVVM75D3ZWJEWMTX3Q, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil> Jul 27 00:26:26.976522761 (2h34m42.757456129s) [noise=true , address=79.127.205.234:4608 , piece_id=W5HB4IJEJKCVH4H2TSDGOUSQMELEYJI4JNSBJG2C6H65K36OQOXA, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil> Jul 27 00:26:27.005071467 (2h34m42.786004845s) [noise=true , address=79.127.205.246:4262 , piece_id=VZNLX55JACROS54O7JQWGBVQCYLRRS2NIBUCTVLL37GGENXO7CHQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil> Jul 27 00:26:27.009050972 (2h34m42.78998441s) [noise=true , address=79.127.205.233:4656 , piece_id=CDINO2ITVWDVXNJ54UYBZME53OJB5R55YA2TYMA7Q5OU77KPUY5Q, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil> Jul 27 00:26:27.823461927 (2h34m43.604395297s) [noise=true , address=64.49.13.140:4508 , piece_id=QFVYOC5QDA5S6STIGOW3LEHJ4QDUB4NOPLBOWNEHYXPAXJYALU5A, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 27 00:26:30.225465377 (2h34m46.006398763s) [noise=true , address=79.127.220.101:4041 , piece_id=DJ4OC2BOSDKKYSHTDV7CPHPYEATJCFYBRNK5O47TRXI2SWOA3YYQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil> Jul 27 00:26:32.566199819 (2h34m48.347133205s) [noise=true , address=45.91.116.9:4277 , piece_id=OHYZ6YEBAXIG4QQ5NPDAMFSMVMUG2AZUQ4HTM5XZVSHDKKKB5OZQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:26:35.104592988 (2h34m50.885526355s) [noise=true , address=77.90.31.19:4875 , piece_id=D7IVBHHMQJKTMRRZWDUOQ6F3DQJ77TMOUC6ZKJOUBRBTRMFWRVGQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil> Jul 27 00:26:38.636390773 (2h34m54.41732415s) [noise=true , address=64.49.13.137:4502 , piece_id=EKYOVB3VGNZEI33TUBBOEETIYNWOXX77HBKRAUQDC3TWU3CPKDYA, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil> Jul 27 00:26:47.765018653 (2h35m3.545952029s) [noise=true , address=79.127.201.215:4360 , piece_id=45AIUENJKBJKZPLJOEMNO2OJVJYE4QHGQDGISWYU63PQSL3HJY2A, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil> Jul 27 00:27:26.279502407 (2h35m42.060435784s) [noise=true , address=77.90.31.17:4793 , piece_id=YPY6GBN42ZLEA5GNKVT774ZY5WZTHLQZHPUJNBLHTCJK2FNHMG4A, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:26.279597697 (2h35m42.060531068s) [noise=true , address=104.36.82.46:4484 , piece_id=REF4XO4UYKOELCCQNRHPRZEBWDYSEZMNU2NMJFLWIDV3LRP6YNKA, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: write tcp 159.223.143.39:39416->104.36.82.46:4484: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:26.279610783 (2h35m42.060544149s) [noise=true , address=77.90.31.16:4746 , piece_id=UDRB4TAYY5Z4PO7ZQFCPS5S7VX3LRQPU4Q5MPFB5C2F5LDI5JWXQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: rpc: dial tcp 77.90.31.16:4746: i/o timeout; conn closed: op Jul 27 00:27:26.279653272 (2h35m42.060586656s) [noise=true , address=104.36.82.52:4109 , piece_id=25MGIJGD3RJMLRNGKIEDS763YLSPR2FF5O5CSXWG2SE6AJL4UYIA, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1hBqAk4HrRmUKRw3RS7xUHZ4rdUMiF25jKWF7jqNrffZG8pEcj, address:104.36.82.52:4109): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:27:26.279734462 (2h35m42.060667843s) [noise=true , address=104.36.82.47:4525 , piece_id=FQALJT36Z7N3V76SRSJWS3WZ3ZIP4736X76SJHHYMMWT2AULLG2A, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: rpc: dial tcp 104.36.82.47:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:27:26.279759917 (2h35m42.060693295s) [seg_pos={0 0}, upload=2377]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:27:26.303763249 (2h35m42.084696614s) [upload=2377]: Done uploading segments... Jul 27 00:27:26.303787289 (2h35m42.084720658s) [upload=2377]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.027458929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXX] duration: 30.031542365s FAIL (2025-07-27 00:27:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.031569043s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............X..X......X.XX....X...X.................XX....X.] duration: 15.083355126s FAIL (2025-07-27 00:26:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.083398316s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......X.X........X...XX...............................X....] duration: 40.449272833s FAIL (2025-07-27 00:26:42 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.449300161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 429.391017ms FAIL (2025-07-27 00:26:39 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 429.504367ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............X..X......X.XX....X...X.................XX....X] duration: 30.001614747s FAIL (2025-07-27 00:26:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001628425s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.03427279s FAIL (2025-07-27 00:26:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== tem_SegmentBegin Jul 27 00:25:26.244624313 (2h33m42.025557709s) [seg_pos={0 0}, upload=2362]: Begin upload segment... Jul 27 00:25:26.244702268 (2h33m42.025635657s) [seg_pos={0 0}, upload=2362]: Done begin upload segment. Jul 27 00:25:26.244742411 (2h33m42.025675798s) [noise=true , address=79.127.201.215:4359 , piece_id=JRUQSYBOAKAA27IJ75A7WQAZABXMCJPDX2WW7B5GOPX54XWFL3WA, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.244800436 (2h33m42.025733816s) [noise=true , address=79.127.220.97:4361 , piece_id=SJB2HOKWSD7XSRM5UZIWSF4QOQUIHZP2MNHSXCWEQCBQT6OXHC6A, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.245202875 (2h33m42.026136251s) [noise=true , address=79.127.205.235:4680 , piece_id=SEWXMVEGRT3HVPTFC4K4I7J6QZTK4PX7OKTYSVOETYPYFOBOC4EA, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.245529665 (2h33m42.026463042s) [noise=true , address=77.90.31.19:4871 , piece_id=36JIQ7Q5PSOIQPSS4GKILZAAZK5UE7W4R7NBY5ZT7F3K2ACSCFRA, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.245608218 (2h33m42.0265416s) [noise=true , address=104.36.82.47:4543 , piece_id=JTI7Z26QAATCQL5HDHZFIUCV22Y7IWYDYTXCDM6YVTN36ZAEDGGQ, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.245899052 (2h33m42.026832435s) [noise=true , address=79.127.205.245:4226 , piece_id=VFIB4IMENPXPMUVNMET7RB37OCN5CCHBLZNQNOUVWBN6DPP34XEQ, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.246176303 (2h33m42.027109689s) [noise=true , address=79.127.219.38:4007 , piece_id=JAOGEBRBWPV7NLELH6NFRZFKC2F7JFTJYTKVW22XEAMOBZFAURNQ, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.246746289 (2h33m42.027679677s) [noise=true , address=79.127.220.98:4422 , piece_id=AGMXWOLUJTM5EBVR4ZP47HWQ76TIQNFRXTYI2RTNJJTOZKZT3YAQ, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.247232536 (2h33m42.028165924s) [noise=true , address=77.90.31.10:4525 , piece_id=JE6O343PIY6ED73P3HIYT7WKIXNDSJ5757GT76JYCB36YCFOJZ4A, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.247682598 (2h33m42.028615984s) [noise=true , address=64.49.13.138:4488 , piece_id=VGFVGLHCM3TEQ3YVSLCIR3NJH6P3RUT3MVXZUHC42EZA7VONKKPQ, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.248106844 (2h33m42.029040232s) [noise=true , address=79.127.205.246:4285 , piece_id=PBA675LKV7GBXPBRLOXKOHBUWLATBOVCLQNNBCXAGBM66RV32JNA, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.248905182 (2h33m42.029838567s) [noise=true , address=104.36.82.55:4223 , piece_id=XBRXK22J6BG7GEBZEOPL44A2OASRFX5NEVLOTJXTTBP2HQSVGG7Q, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.249296727 (2h33m42.030230108s) [noise=true , address=77.90.31.12:4599 , piece_id=NNL6O4W7IA6EK2GPK4FENKNLE4NGCH3CL7OCO6NGDI642QQLNEHA, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.250505895 (2h33m42.031439283s) [noise=true , address=77.90.31.15:4730 , piece_id=R3FN2M5SVCRB56DNTJVLFYZXQ6VRITBMT6ZN4TSJRZVDJ63VYYTA, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.251100881 (2h33m42.03203427s) [noise=true , address=104.36.82.3:4782 , piece_id=6CZYKHGWCWM6XYO36F6OIUUTJDQO5XLQ7RBVJEOF2MWRX7WVBVGA, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.251578633 (2h33m42.032512018s) [noise=true , address=77.90.31.13:4629 , piece_id=UMN4LM75OD64IKS66CGI2VQQ2YYF7QAF336JMGHGHEK74HZJG63A, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.252157696 (2h33m42.033091108s) [noise=true , address=77.90.31.16:4765 , piece_id=UWU3X4UEF6UZBCMZJINY2V3JIY2GYQVXHYNHFOTHYQNBPLSHGKEQ, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.252514978 (2h33m42.03344836s) [noise=true , address=104.36.82.52:4091 , piece_id=CJO2M5QTH73G5AK3P72JE5SBJWLGEMQZAJ4C7CAVH2TAG2F56ODQ, seg_pos={0 0}, upload=2362]: Uploading piece... Jul 27 00:25:26.291604267 (2h33m42.072537643s) [noise=true , address=77.90.31.12:4599 , piece_id=NNL6O4W7IA6EK2GPK4FENKNLE4NGCH3CL7OCO6NGDI642QQLNEHA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: storage node overloaded, request limit: 2000 Jul 27 00:25:26.329006714 (2h33m42.109940097s) [upload=2362]: Next returned nil segment Jul 27 00:25:26.329028392 (2h33m42.109961773s) [upload=2362]: Waiting for error group managing segments... Jul 27 00:25:26.353992946 (2h33m42.134926333s) [noise=true , address=79.127.219.38:4007 , piece_id=JAOGEBRBWPV7NLELH6NFRZFKC2F7JFTJYTKVW22XEAMOBZFAURNQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil> Jul 27 00:25:26.367473401 (2h33m42.148406783s) [noise=true , address=77.90.31.15:4730 , piece_id=R3FN2M5SVCRB56DNTJVLFYZXQ6VRITBMT6ZN4TSJRZVDJ63VYYTA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil> Jul 27 00:25:26.837312215 (2h33m42.618245598s) [noise=true , address=79.127.220.97:4361 , piece_id=SJB2HOKWSD7XSRM5UZIWSF4QOQUIHZP2MNHSXCWEQCBQT6OXHC6A, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil> Jul 27 00:25:26.853999386 (2h33m42.634932748s) [noise=true , address=79.127.205.246:4285 , piece_id=PBA675LKV7GBXPBRLOXKOHBUWLATBOVCLQNNBCXAGBM66RV32JNA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil> Jul 27 00:25:26.872098702 (2h33m42.653032071s) [noise=true , address=79.127.205.245:4226 , piece_id=VFIB4IMENPXPMUVNMET7RB37OCN5CCHBLZNQNOUVWBN6DPP34XEQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil> Jul 27 00:25:27.036596515 (2h33m42.817529893s) [noise=true , address=79.127.205.235:4680 , piece_id=SEWXMVEGRT3HVPTFC4K4I7J6QZTK4PX7OKTYSVOETYPYFOBOC4EA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil> Jul 27 00:25:29.524653177 (2h33m45.305586562s) [noise=true , address=79.127.220.98:4422 , piece_id=AGMXWOLUJTM5EBVR4ZP47HWQ76TIQNFRXTYI2RTNJJTOZKZT3YAQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil> Jul 27 00:25:29.826180626 (2h33m45.607113999s) [noise=true , address=77.90.31.19:4871 , piece_id=36JIQ7Q5PSOIQPSS4GKILZAAZK5UE7W4R7NBY5ZT7F3K2ACSCFRA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil> Jul 27 00:25:35.766937900 (2h33m51.547871263s) [noise=true , address=64.49.13.138:4488 , piece_id=VGFVGLHCM3TEQ3YVSLCIR3NJH6P3RUT3MVXZUHC42EZA7VONKKPQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil> Jul 27 00:25:56.122288933 (2h34m11.903222302s) [noise=true , address=79.127.201.215:4359 , piece_id=JRUQSYBOAKAA27IJ75A7WQAZABXMCJPDX2WW7B5GOPX54XWFL3WA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil> Jul 27 00:26:06.135670734 (2h34m21.916604104s) [noise=true , address=77.90.31.13:4629 , piece_id=UMN4LM75OD64IKS66CGI2VQQ2YYF7QAF336JMGHGHEK74HZJG63A, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil> Jul 27 00:26:26.168818382 (2h34m41.949751759s) [noise=true , address=104.36.82.52:4091 , piece_id=CJO2M5QTH73G5AK3P72JE5SBJWLGEMQZAJ4C7CAVH2TAG2F56ODQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:26.168898636 (2h34m41.949832015s) [noise=true , address=104.36.82.47:4543 , piece_id=JTI7Z26QAATCQL5HDHZFIUCV22Y7IWYDYTXCDM6YVTN36ZAEDGGQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: write tcp 159.223.143.39:47730->104.36.82.47:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:26.169077250 (2h34m41.950010628s) [noise=true , address=104.36.82.55:4223 , piece_id=XBRXK22J6BG7GEBZEOPL44A2OASRFX5NEVLOTJXTTBP2HQSVGG7Q, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:104.36.82.55:4223): protocol: write tcp 159.223.143.39:38228->104.36.82.55:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:26.169121767 (2h34m41.950055133s) [noise=true , address=104.36.82.3:4782 , piece_id=6CZYKHGWCWM6XYO36F6OIUUTJDQO5XLQ7RBVJEOF2MWRX7WVBVGA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:104.36.82.3:4782): protocol: write tcp 159.223.143.39:54194->104.36.82.3:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:26.169179051 (2h34m41.950112426s) [noise=true , address=77.90.31.10:4525 , piece_id=JE6O343PIY6ED73P3HIYT7WKIXNDSJ5757GT76JYCB36YCFOJZ4A, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:26.169184440 (2h34m41.95011782s) [noise=true , address=77.90.31.16:4765 , piece_id=UWU3X4UEF6UZBCMZJINY2V3JIY2GYQVXHYNHFOTHYQNBPLSHGKEQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:12CSrtv2fMmFBx9zje9zBevpVzxqyhH7fL7VnHbBFEYFRPuwCKK, address:77.90.31.16:4765): protocol: write tcp 159.223.143.39:48810->77.90.31.16:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:26:26.169209994 (2h34m41.950143378s) [seg_pos={0 0}, upload=2362]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:26:26.200843887 (2h34m41.981777265s) [upload=2362]: Done uploading segments... Jul 27 00:26:26.200875323 (2h34m41.981808701s) [upload=2362]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.034294346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 181.275359ms FAIL (2025-07-27 00:26:20 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 181.311348ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX.] duration: 2.489898626s FAIL (2025-07-27 00:26:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 5.226875599 (2h33m1.007808984s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:35016->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 27 00:24:45.226924742 (2h33m1.007858123s) [seg_pos={0 0}, upload=2341]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:24:45.260403710 (2h33m1.041337086s) [upload=2341]: Done uploading segments... Jul 27 00:24:45.260436570 (2h33m1.041369954s) [upload=2341]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:25:13.130330845 (2h33m28.911264233s) [upload=2358]: Starting upload Jul 27 00:25:13.130427709 (2h33m28.911361097s) [upload=2358]: Done starting upload Jul 27 00:25:13.130504194 (2h33m28.911437572s) [upload=2358]: Uploading segments... Jul 27 00:25:13.130768741 (2h33m28.911702127s) [upload=2358]: Got next segment. Inline: false Jul 27 00:25:13.130825048 (2h33m28.911758432s) [upload=2358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:25:13.130832706 (2h33m28.911766093s) [upload=2358]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:25:13.156703675 (2h33m28.937637054s) [seg_pos={0 0}, upload=2358]: Begin upload segment... Jul 27 00:25:13.156790074 (2h33m28.937723451s) [seg_pos={0 0}, upload=2358]: Done begin upload segment. Jul 27 00:25:13.156839061 (2h33m28.937772446s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.156878026 (2h33m28.937811413s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157292455 (2h33m28.938225837s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157494201 (2h33m28.938427581s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157651501 (2h33m28.938584887s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158080485 (2h33m28.939013869s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158127600 (2h33m28.939060987s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158413346 (2h33m28.939346724s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158633735 (2h33m28.939567118s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158790022 (2h33m28.939723394s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.159238365 (2h33m28.940171753s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.160717151 (2h33m28.941650518s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.161474674 (2h33m28.942408056s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.162300982 (2h33m28.943234369s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.163746259 (2h33m28.94467965s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.164391211 (2h33m28.945324597s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.286660997 (2h33m29.067594379s) [upload=2358]: Next returned nil segment Jul 27 00:25:13.286684184 (2h33m29.067617547s) [upload=2358]: Waiting for error group managing segments... Jul 27 00:25:13.292602587 (2h33m29.073535969s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.292748204 (2h33m29.073681574s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.293326557 (2h33m29.074259939s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.293361832 (2h33m29.074295202s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.294304357 (2h33m29.075237733s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.295589750 (2h33m29.076523117s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.295634225 (2h33m29.076567588s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.296581902 (2h33m29.077515282s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.297185145 (2h33m29.078118508s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.325234980 (2h33m29.106168359s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.360841866 (2h33m29.141775245s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:43.100924665 (2h33m58.88185803s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 159.223.143.39:56852->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101183051 (2h33m58.882116415s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:25:43.101557492 (2h33m58.882490858s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101690637 (2h33m58.882624007s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:25:43.101784731 (2h33m58.882718095s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:54568->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101803464 (2h33m58.882736842s) [seg_pos={0 0}, upload=2358]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:25:43.126349574 (2h33m58.90728296s) [upload=2358]: Done uploading segments... Jul 27 00:25:43.126382341 (2h33m58.907315726s) [upload=2358]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.489926749s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XX] duration: 30.028556323s FAIL (2025-07-27 00:26:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.028648049s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........X.X........X...XX...............................X...] duration: 46.265522184s FAIL (2025-07-27 00:25:53 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.265558064s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............X..X......X.XX....X...X.................XX....] duration: 2.562520385s FAIL (2025-07-27 00:25:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.562546563s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX] duration: 30.027403278s FAIL (2025-07-27 00:25:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 5.226875599 (2h33m1.007808984s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:35016->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 27 00:24:45.226924742 (2h33m1.007858123s) [seg_pos={0 0}, upload=2341]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:24:45.260403710 (2h33m1.041337086s) [upload=2341]: Done uploading segments... Jul 27 00:24:45.260436570 (2h33m1.041369954s) [upload=2341]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:25:13.130330845 (2h33m28.911264233s) [upload=2358]: Starting upload Jul 27 00:25:13.130427709 (2h33m28.911361097s) [upload=2358]: Done starting upload Jul 27 00:25:13.130504194 (2h33m28.911437572s) [upload=2358]: Uploading segments... Jul 27 00:25:13.130768741 (2h33m28.911702127s) [upload=2358]: Got next segment. Inline: false Jul 27 00:25:13.130825048 (2h33m28.911758432s) [upload=2358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:25:13.130832706 (2h33m28.911766093s) [upload=2358]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:25:13.156703675 (2h33m28.937637054s) [seg_pos={0 0}, upload=2358]: Begin upload segment... Jul 27 00:25:13.156790074 (2h33m28.937723451s) [seg_pos={0 0}, upload=2358]: Done begin upload segment. Jul 27 00:25:13.156839061 (2h33m28.937772446s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.156878026 (2h33m28.937811413s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157292455 (2h33m28.938225837s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157494201 (2h33m28.938427581s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.157651501 (2h33m28.938584887s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158080485 (2h33m28.939013869s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158127600 (2h33m28.939060987s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158413346 (2h33m28.939346724s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158633735 (2h33m28.939567118s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.158790022 (2h33m28.939723394s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.159238365 (2h33m28.940171753s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.160717151 (2h33m28.941650518s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.161474674 (2h33m28.942408056s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.162300982 (2h33m28.943234369s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.163746259 (2h33m28.94467965s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.164391211 (2h33m28.945324597s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Uploading piece... Jul 27 00:25:13.286660997 (2h33m29.067594379s) [upload=2358]: Next returned nil segment Jul 27 00:25:13.286684184 (2h33m29.067617547s) [upload=2358]: Waiting for error group managing segments... Jul 27 00:25:13.292602587 (2h33m29.073535969s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.292748204 (2h33m29.073681574s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.293326557 (2h33m29.074259939s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.293361832 (2h33m29.074295202s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.294304357 (2h33m29.075237733s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.295589750 (2h33m29.076523117s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.295634225 (2h33m29.076567588s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.296581902 (2h33m29.077515282s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.297185145 (2h33m29.078118508s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.325234980 (2h33m29.106168359s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:13.360841866 (2h33m29.141775245s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil> Jul 27 00:25:43.100924665 (2h33m58.88185803s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 159.223.143.39:56852->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101183051 (2h33m58.882116415s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:25:43.101557492 (2h33m58.882490858s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101690637 (2h33m58.882624007s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:25:43.101784731 (2h33m58.882718095s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:54568->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:43.101803464 (2h33m58.882736842s) [seg_pos={0 0}, upload=2358]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:25:43.126349574 (2h33m58.90728296s) [upload=2358]: Done uploading segments... Jul 27 00:25:43.126382341 (2h33m58.907315726s) [upload=2358]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.02743408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029152973s FAIL (2025-07-27 00:25:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== DOTA, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.144717031 (2h32m41.925650412s) [noise=true , address=79.127.205.233:4646 , piece_id=MVFEMCQGCRD66M2354ZI4PT6XKCGGASCZ32ETPZ2R6FRW5HP7ICA, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.145138585 (2h32m41.926071965s) [noise=true , address=104.36.82.55:4217 , piece_id=2CD4YAEUJEXVEE4DE77CNVMQ6FIO3D6KSZE4NGK44LKF4LN5PBIA, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.145255059 (2h32m41.926188438s) [noise=true , address=77.90.31.21:4882 , piece_id=IXQM5LKLCCZVTI43446VNZQMGDZOG6I27OUIFFWKN7COEIPLJYGA, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.145500697 (2h32m41.926434074s) [noise=true , address=104.36.82.46:4495 , piece_id=CINFJ5JW56RNUMXAW2GKNFQJFJZVFN3ZIYFSXY2UEBAW62KDCFHA, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.145725984 (2h32m41.926659369s) [noise=true , address=104.36.82.47:4528 , piece_id=BUNIMSFRCCNX3XF6NROPGZLNRKLLQCTIQHHTI4IMINZUJWOCMDVQ, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.145859916 (2h32m41.926793285s) [noise=true , address=79.127.220.101:4048 , piece_id=RAHM5QA5YM4YDONCLCBHTUAE2HSIBDGS3DTYIXEPLJSVYWU4W2VA, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.146193690 (2h32m41.927127078s) [noise=true , address=77.90.31.11:4573 , piece_id=TCL6SVCEE72Q5L3C4AP3P5YIDOXRFR3L7V4V5T4GB75LFUQHS52Q, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.146672841 (2h32m41.927606226s) [noise=true , address=104.36.82.54:4161 , piece_id=64PWUSYXKZ2NAWMXHOI3VSG3N3A32PMSF4H5GLLVZJSN3KH6DQSA, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.148131952 (2h32m41.92906534s) [noise=true , address=77.90.31.12:4597 , piece_id=R22R3IALNL52S27QFAHBAWYGK7AG5W2MK7ZE5O66AHWS2GAOG4QQ, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.148644203 (2h32m41.929577591s) [noise=true , address=79.127.205.245:4226 , piece_id=BGDKLU4H4UVGF23EVXUJMFKLQBFFFZWUDSYRKHNSLXYACJTHLAKA, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.149154530 (2h32m41.930087912s) [noise=true , address=79.127.205.246:4283 , piece_id=TDQ5XREXHDRRXQ2SNAEFVLSPPE2NUU3LM7GPC7OKM36HSHWIITNA, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.149164538 (2h32m41.930097909s) [noise=true , address=77.90.31.15:4723 , piece_id=2CSXXBCMY6NGHRNFSVI472IAHBWRA4AQDTWHWQMTOG5JSBSDYQDQ, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.149632311 (2h32m41.930565704s) [noise=true , address=104.36.82.53:4141 , piece_id=FTW4KKWZC3CXQEAQSPEJEZFYIR4IVR63LN5SXEITGJEBIBINNYTQ, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.150650371 (2h32m41.931583759s) [noise=true , address=79.127.201.215:4359 , piece_id=I346M37ZESQ546M46GVTY55U56W6ZG3LAAQHIQOATN64RO6KG7IA, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.151188549 (2h32m41.932121935s) [noise=true , address=77.90.31.16:4764 , piece_id=OROXSACR2VZOSI7BGUCMAYZVUSOS3PZXNCNCMBGC22XHVLI4LJNQ, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.152270751 (2h32m41.933204131s) [noise=true , address=77.90.31.18:4825 , piece_id=PZISWJGVHM2DQWW7ZA6IB7FEYCNRF44TKKZTZL2BRGLDSD2Z623A, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.152829565 (2h32m41.933762954s) [noise=true , address=79.127.219.38:4006 , piece_id=OI2SLC56PJXOMDVF7LSGGEZ5DAQ5EP32L76BCPO3OF2AQAWM3GIQ, seg_pos={0 0}, upload=2345]: Uploading piece... Jul 27 00:24:26.288975838 (2h32m42.069909337s) [upload=2345]: Next returned nil segment Jul 27 00:24:26.288996776 (2h32m42.069930166s) [upload=2345]: Waiting for error group managing segments... Jul 27 00:24:26.291729785 (2h32m42.072663165s) [noise=true , address=79.127.220.101:4048 , piece_id=RAHM5QA5YM4YDONCLCBHTUAE2HSIBDGS3DTYIXEPLJSVYWU4W2VA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:24:26.304405017 (2h32m42.085338401s) [noise=true , address=79.127.219.38:4006 , piece_id=OI2SLC56PJXOMDVF7LSGGEZ5DAQ5EP32L76BCPO3OF2AQAWM3GIQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil> Jul 27 00:24:26.309698318 (2h32m42.090631683s) [noise=true , address=77.90.31.15:4723 , piece_id=2CSXXBCMY6NGHRNFSVI472IAHBWRA4AQDTWHWQMTOG5JSBSDYQDQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil> Jul 27 00:24:26.340222509 (2h32m42.12115588s) [noise=true , address=77.90.31.21:4882 , piece_id=IXQM5LKLCCZVTI43446VNZQMGDZOG6I27OUIFFWKN7COEIPLJYGA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil> Jul 27 00:24:26.361893801 (2h32m42.142827179s) [noise=true , address=77.90.31.18:4825 , piece_id=PZISWJGVHM2DQWW7ZA6IB7FEYCNRF44TKKZTZL2BRGLDSD2Z623A, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil> Jul 27 00:24:26.752522993 (2h32m42.533456358s) [noise=true , address=79.127.205.246:4283 , piece_id=TDQ5XREXHDRRXQ2SNAEFVLSPPE2NUU3LM7GPC7OKM36HSHWIITNA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil> Jul 27 00:24:26.779164681 (2h32m42.560098057s) [noise=true , address=79.127.205.233:4646 , piece_id=MVFEMCQGCRD66M2354ZI4PT6XKCGGASCZ32ETPZ2R6FRW5HP7ICA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil> Jul 27 00:24:26.779792648 (2h32m42.560726017s) [noise=true , address=79.127.205.245:4226 , piece_id=BGDKLU4H4UVGF23EVXUJMFKLQBFFFZWUDSYRKHNSLXYACJTHLAKA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil> Jul 27 00:24:27.345648921 (2h32m43.126582282s) [noise=true , address=77.90.31.12:4597 , piece_id=R22R3IALNL52S27QFAHBAWYGK7AG5W2MK7ZE5O66AHWS2GAOG4QQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil> Jul 27 00:24:49.617145628 (2h33m5.398078998s) [noise=true , address=79.127.201.215:4359 , piece_id=I346M37ZESQ546M46GVTY55U56W6ZG3LAAQHIQOATN64RO6KG7IA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil> Jul 27 00:25:26.074779638 (2h33m41.855713013s) [noise=true , address=104.36.82.55:4217 , piece_id=2CD4YAEUJEXVEE4DE77CNVMQ6FIO3D6KSZE4NGK44LKF4LN5PBIA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:12bw9V8UMi7pFxqJ8fh22x5ZPmtJUJyJGw8hD9NhQcfHdfx4cZj, address:104.36.82.55:4217): protocol: rpc: dial tcp 104.36.82.55:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:26.074846936 (2h33m41.855780311s) [noise=true , address=104.36.82.47:4528 , piece_id=BUNIMSFRCCNX3XF6NROPGZLNRKLLQCTIQHHTI4IMINZUJWOCMDVQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: rpc: dial tcp 104.36.82.47:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:26.074924665 (2h33m41.855858027s) [noise=true , address=77.90.31.11:4573 , piece_id=TCL6SVCEE72Q5L3C4AP3P5YIDOXRFR3L7V4V5T4GB75LFUQHS52Q, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV, address:77.90.31.11:4573): protocol: read tcp 159.223.143.39:50386->77.90.31.11:4573: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:26.074968338 (2h33m41.855901715s) [noise=true , address=77.90.31.16:4764 , piece_id=OROXSACR2VZOSI7BGUCMAYZVUSOS3PZXNCNCMBGC22XHVLI4LJNQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, address:77.90.31.16:4764): protocol: rpc: dial tcp 77.90.31.16:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:26.074981473 (2h33m41.85591485s) [noise=true , address=104.36.82.46:4495 , piece_id=CINFJ5JW56RNUMXAW2GKNFQJFJZVFN3ZIYFSXY2UEBAW62KDCFHA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:1YpHarCJ3ouMGTs59udhU7az9h68ELEHTa2aTAax2chR2Gux1x, address:104.36.82.46:4495): protocol: rpc: dial tcp 104.36.82.46:4495: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:26.075130480 (2h33m41.856063857s) [noise=true , address=104.36.82.54:4161 , piece_id=64PWUSYXKZ2NAWMXHOI3VSG3N3A32PMSF4H5GLLVZJSN3KH6DQSA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD, address:104.36.82.54:4161): protocol: rpc: dial tcp 104.36.82.54:4161: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:26.075179827 (2h33m41.856113192s) [noise=true , address=104.36.82.52:4101 , piece_id=KJAT3DZY43IUD2X7VRFL5KTLTFKPYFBTH7IX2HSMBAPRFUTXDOTA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: write tcp 159.223.143.39:47750->104.36.82.52:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:26.075198465 (2h33m41.856131826s) [noise=true , address=104.36.82.53:4141 , piece_id=FTW4KKWZC3CXQEAQSPEJEZFYIR4IVR63LN5SXEITGJEBIBINNYTQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:25:26.075236070 (2h33m41.856169451s) [seg_pos={0 0}, upload=2345]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:25:26.100957851 (2h33m41.881891231s) [upload=2345]: Done uploading segments... Jul 27 00:25:26.100988962 (2h33m41.881922344s) [upload=2345]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.029205696s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.X] duration: 30.032190265s FAIL (2025-07-27 00:25:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.032224868s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...............X..X......X.XX....X...X.................XX...] duration: 10.516559351s FAIL (2025-07-27 00:24:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.516593906s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXX] duration: 30.037846199s FAIL (2025-07-27 00:24:45 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 75): protocol: write tcp 159.223.143.39:53440->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:23:35.802082296 (2h31m51.583015681s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:23:35.830966119 (2h31m51.611899487s) [upload=2325]: Done uploading segments... Jul 27 00:23:35.830993756 (2h31m51.611927137s) [upload=2325]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 27 00:24:15.256062013 (2h32m31.036995375s) [upload=2341]: Starting upload Jul 27 00:24:15.256158699 (2h32m31.037092079s) [upload=2341]: Done starting upload Jul 27 00:24:15.256302903 (2h32m31.037236284s) [upload=2341]: Uploading segments... Jul 27 00:24:15.256454993 (2h32m31.037388363s) [upload=2341]: Got next segment. Inline: false Jul 27 00:24:15.256508209 (2h32m31.037441584s) [upload=2341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:24:15.256513899 (2h32m31.037447274s) [upload=2341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:24:15.290479823 (2h32m31.071413197s) [seg_pos={0 0}, upload=2341]: Begin upload segment... Jul 27 00:24:15.290522174 (2h32m31.071455556s) [seg_pos={0 0}, upload=2341]: Done begin upload segment. Jul 27 00:24:15.290568944 (2h32m31.071502329s) [noise=true , address=77.90.31.14:4661 , piece_id=354M4K53NIFHG2F5B34YXTAAXVOLXCCCH4SHAE56MPI7HZ33GGDA, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.290880163 (2h32m31.071813545s) [noise=true , address=77.90.31.16:4735 , piece_id=F6TKVMHOYXIRQJ2TIALFJGWBD3KAJP3436JXWQ6ACHYEJ5ARNKJA, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.291096947 (2h32m31.072030331s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.291376582 (2h32m31.072309958s) [noise=true , address=45.91.116.8:4047 , piece_id=KQAQDBYYPTKLQVHWPFAQPMQAQT3LVG2ZMIZIUT27OVKZID3QA3CQ, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.291546777 (2h32m31.072480163s) [noise=true , address=77.90.31.13:4628 , piece_id=Q6LXQVYH5NLRUHPVR7GEM435RQYJA2XYZYLXRN5CMEOMXXU4B2MQ, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.291737132 (2h32m31.072670515s) [noise=true , address=77.90.31.10:4519 , piece_id=ES7E6WWHHSYR4ZFJ3U56UA2C4ID6E2UO3UTBQMK3MHWXKKF7NTSA, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.292151224 (2h32m31.073084609s) [noise=true , address=45.91.116.12:4226 , piece_id=5VLTPXGXLV3AVI4IRGFEHPSAEFYGCKRZWLUBQA5OK4NQKQQB7FZQ, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.292277278 (2h32m31.073210659s) [noise=true , address=104.36.82.7:4911 , piece_id=FATE6O34GWR4OFDNN3MJWFA2DKTFJF4O7TS3EEVBHC6PYBIPWZDA, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.292593888 (2h32m31.073527263s) [noise=true , address=45.91.116.6:4973 , piece_id=3JKFXNCOWGZDS5DY5RPU2G32DLRY2OAM6KGVTQ6NXO5ENZKYLE2Q, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.292661711 (2h32m31.073595092s) [noise=true , address=45.91.116.13:4154 , piece_id=CMWMJACSKS6XRRCGE3JIVKJ56C7HQMYP7MK3F4OZPQ4XRUL33GUQ, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.292906506 (2h32m31.073839884s) [noise=true , address=45.91.116.5:4297 , piece_id=QHJFDHCZZ7SS5JP3X6TA2DDVNBRL4CX2FT7BFH6S5IWBSKYZ7LNQ, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.293095594 (2h32m31.074028973s) [noise=true , address=45.91.116.9:4261 , piece_id=2Q7BMHN3ZXMRVGWHXTYA7PPQBH4CZ4PKH2EZA6Y3IBJJFP2COH3A, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.293278492 (2h32m31.074211875s) [noise=true , address=45.91.116.4:4937 , piece_id=2VLKTZW4B2BUNWFOGMLRD5NLWAMCA35J2FOMM5GHRPRFTIVOFTIA, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.293527724 (2h32m31.074461109s) [noise=true , address=104.36.82.6:4876 , piece_id=OMCJOJ3BRXAIDOCT4ONBGM7WLMAESXHCM7FRQGUNQSEQ4KNXGFJA, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.293665152 (2h32m31.074598528s) [noise=true , address=77.90.31.10:4518 , piece_id=LGVDHB2LJLE7X3ZQXRAEADD4EOM5XAQJLRX3K3YSQGIMKW7ULFRA, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.293996613 (2h32m31.074929991s) [noise=true , address=45.91.116.7:4009 , piece_id=33K7VJVNW6R3DPD34ASRJQMWN5CFNLBDUTGCBMPKMW7OUZK2VPDQ, seg_pos={0 0}, upload=2341]: Uploading piece... Jul 27 00:24:15.377667276 (2h32m31.158600652s) [upload=2341]: Next returned nil segment Jul 27 00:24:15.377684557 (2h32m31.158617932s) [upload=2341]: Waiting for error group managing segments... Jul 27 00:24:15.381462128 (2h32m31.162395496s) [noise=true , address=45.91.116.6:4973 , piece_id=3JKFXNCOWGZDS5DY5RPU2G32DLRY2OAM6KGVTQ6NXO5ENZKYLE2Q, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil> Jul 27 00:24:15.382306088 (2h32m31.163239457s) [noise=true , address=45.91.116.12:4226 , piece_id=5VLTPXGXLV3AVI4IRGFEHPSAEFYGCKRZWLUBQA5OK4NQKQQB7FZQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil> Jul 27 00:24:15.382553700 (2h32m31.163487072s) [noise=true , address=45.91.116.9:4261 , piece_id=2Q7BMHN3ZXMRVGWHXTYA7PPQBH4CZ4PKH2EZA6Y3IBJJFP2COH3A, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil> Jul 27 00:24:15.382934775 (2h32m31.163868154s) [noise=true , address=45.91.116.13:4154 , piece_id=CMWMJACSKS6XRRCGE3JIVKJ56C7HQMYP7MK3F4OZPQ4XRUL33GUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil> Jul 27 00:24:15.383616941 (2h32m31.164550309s) [noise=true , address=45.91.116.5:4297 , piece_id=QHJFDHCZZ7SS5JP3X6TA2DDVNBRL4CX2FT7BFH6S5IWBSKYZ7LNQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil> Jul 27 00:24:15.384175789 (2h32m31.165109182s) [noise=true , address=45.91.116.8:4047 , piece_id=KQAQDBYYPTKLQVHWPFAQPMQAQT3LVG2ZMIZIUT27OVKZID3QA3CQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil> Jul 27 00:24:15.384717162 (2h32m31.165650542s) [noise=true , address=77.90.31.14:4661 , piece_id=354M4K53NIFHG2F5B34YXTAAXVOLXCCCH4SHAE56MPI7HZ33GGDA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil> Jul 27 00:24:15.385236419 (2h32m31.16616979s) [noise=true , address=45.91.116.7:4009 , piece_id=33K7VJVNW6R3DPD34ASRJQMWN5CFNLBDUTGCBMPKMW7OUZK2VPDQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil> Jul 27 00:24:15.385764587 (2h32m31.166697956s) [noise=true , address=45.91.116.4:4937 , piece_id=2VLKTZW4B2BUNWFOGMLRD5NLWAMCA35J2FOMM5GHRPRFTIVOFTIA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil> Jul 27 00:24:16.802561849 (2h32m32.583495218s) [noise=true , address=77.90.31.13:4628 , piece_id=Q6LXQVYH5NLRUHPVR7GEM435RQYJA2XYZYLXRN5CMEOMXXU4B2MQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil> Jul 27 00:24:45.226162479 (2h33m1.007095865s) [noise=true , address=77.90.31.10:4519 , piece_id=ES7E6WWHHSYR4ZFJ3U56UA2C4ID6E2UO3UTBQMK3MHWXKKF7NTSA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: rpc: dial tcp 77.90.31.10:4519: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:45.226264594 (2h33m1.007197978s) [noise=true , address=104.36.82.7:4911 , piece_id=FATE6O34GWR4OFDNN3MJWFA2DKTFJF4O7TS3EEVBHC6PYBIPWZDA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:45.226335397 (2h33m1.007268781s) [noise=true , address=104.36.82.6:4876 , piece_id=OMCJOJ3BRXAIDOCT4ONBGM7WLMAESXHCM7FRQGUNQSEQ4KNXGFJA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:54796->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:45.226417188 (2h33m1.007350565s) [noise=true , address=77.90.31.10:4518 , piece_id=LGVDHB2LJLE7X3ZQXRAEADD4EOM5XAQJLRX3K3YSQGIMKW7ULFRA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:45.226515017 (2h33m1.007448393s) [noise=true , address=77.90.31.16:4735 , piece_id=F6TKVMHOYXIRQJ2TIALFJGWBD3KAJP3436JXWQ6ACHYEJ5ARNKJA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 159.223.143.39:35544->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:24:45.226875599 (2h33m1.007808984s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:35016->104.36.82.4:4801: use of closed network connection; context deadline exceeded Jul 27 00:24:45.226924742 (2h33m1.007858123s) [seg_pos={0 0}, upload=2341]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:24:45.260403710 (2h33m1.041337086s) [upload=2341]: Done uploading segments... Jul 27 00:24:45.260436570 (2h33m1.041369954s) [upload=2341]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.037870914s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)