Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 571.915234ms FAIL (2025-07-26 20:58:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 571.944475ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 765.777285ms FAIL (2025-07-26 20:58:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 765.810824ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 332.937642ms FAIL (2025-07-26 20:58:35 UTC) orbiter warning: uplink: too many requests Duration: 333.044386ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 279.91624ms FAIL (2025-07-26 20:58:19 UTC) orbiter warning: uplink: too many requests Duration: 279.944528ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 768.570821ms FAIL (2025-07-26 20:58:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 768.598109ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 779.729391ms FAIL (2025-07-26 20:58:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 779.758686ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 613.505985ms FAIL (2025-07-26 20:57:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 613.56116ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 732.766004ms FAIL (2025-07-26 20:57:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 732.793728ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 490.246446ms FAIL (2025-07-26 20:57:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 490.275172ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 876.459118ms FAIL (2025-07-26 20:57:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 876.488248ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 420.993825ms FAIL (2025-07-26 20:57:22 UTC) orbiter warning: uplink: too many requests Duration: 421.022728ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 420.60924ms FAIL (2025-07-26 20:56:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 420.636282ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 523.62812ms FAIL (2025-07-26 20:56:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 523.648604ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 978.565629ms FAIL (2025-07-26 20:56:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 978.586358ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 253.714696ms FAIL (2025-07-26 20:56:31 UTC) orbiter warning: uplink: too many requests Duration: 253.825696ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 1.068775167s FAIL (2025-07-26 20:56:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.06880802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 792.81081ms FAIL (2025-07-26 20:56:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 792.838174ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 632.78897ms FAIL (2025-07-26 20:55:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 632.815519ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 452.450722ms FAIL (2025-07-26 20:55:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 452.493072ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 537.545269ms FAIL (2025-07-26 20:55:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 537.572975ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 205.931026ms FAIL (2025-07-26 20:55:31 UTC) orbiter warning: uplink: too many requests Duration: 205.964479ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 1.275211988s FAIL (2025-07-26 20:55:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.275643827s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 1.307571489s FAIL (2025-07-26 20:55:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.307598139s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 423.846593ms FAIL (2025-07-26 20:54:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 423.880215ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 399.82469ms FAIL (2025-07-26 20:54:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 399.853426ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 198.323333ms FAIL (2025-07-26 20:54:46 UTC) orbiter warning: uplink: too many requests Duration: 198.355981ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 617.595325ms FAIL (2025-07-26 20:54:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 617.64021ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 941.738647ms FAIL (2025-07-26 20:54:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 941.768919ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 245.383976ms FAIL (2025-07-26 20:54:13 UTC) orbiter warning: uplink: too many requests Duration: 245.418177ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 462.389182ms FAIL (2025-07-26 20:54:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 462.409083ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 729.861205ms FAIL (2025-07-26 20:54:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 729.884993ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 211.805098ms FAIL (2025-07-26 20:53:57 UTC) orbiter warning: uplink: too many requests Duration: 211.830096ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 216.925752ms FAIL (2025-07-26 20:53:27 UTC) orbiter warning: uplink: too many requests Duration: 216.954064ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 1.851455546s FAIL (2025-07-26 20:53:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.851486203s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 473.688288ms FAIL (2025-07-26 20:53:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 473.723048ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 1.230765913s FAIL (2025-07-26 20:53:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.230796875s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 498.066824ms FAIL (2025-07-26 20:52:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 498.097554ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 538.186424ms FAIL (2025-07-26 20:52:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 538.220315ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 423.125681ms FAIL (2025-07-26 20:52:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 423.154999ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 224.771788ms FAIL (2025-07-26 20:52:25 UTC) orbiter warning: uplink: too many requests Duration: 224.801982ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 472.399652ms FAIL (2025-07-26 20:52:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 472.430898ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 1.342549268s FAIL (2025-07-26 20:52:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.342602066s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 807.090605ms FAIL (2025-07-26 20:51:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 807.119924ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 446.855199ms FAIL (2025-07-26 20:51:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.875455ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 234.742297ms FAIL (2025-07-26 20:51:45 UTC) orbiter warning: uplink: too many requests Duration: 234.769205ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 703.181996ms FAIL (2025-07-26 20:51:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 703.20679ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 516.449946ms FAIL (2025-07-26 20:50:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 516.47677ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 225.823161ms FAIL (2025-07-26 20:50:52 UTC) orbiter warning: uplink: too many requests Duration: 225.869004ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 476.956503ms FAIL (2025-07-26 20:50:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 476.977418ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 648.001555ms FAIL (2025-07-26 20:50:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 648.044234ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)