Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 1.230765913s FAIL (2025-07-26 20:53:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.230796875s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 498.066824ms FAIL (2025-07-26 20:52:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 498.097554ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 538.186424ms FAIL (2025-07-26 20:52:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 538.220315ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 423.125681ms FAIL (2025-07-26 20:52:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 423.154999ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 224.771788ms FAIL (2025-07-26 20:52:25 UTC) orbiter warning: uplink: too many requests Duration: 224.801982ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 472.399652ms FAIL (2025-07-26 20:52:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 472.430898ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 1.342549268s FAIL (2025-07-26 20:52:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.342602066s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 807.090605ms FAIL (2025-07-26 20:51:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 807.119924ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 446.855199ms FAIL (2025-07-26 20:51:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.875455ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 234.742297ms FAIL (2025-07-26 20:51:45 UTC) orbiter warning: uplink: too many requests Duration: 234.769205ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 703.181996ms FAIL (2025-07-26 20:51:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 703.20679ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 516.449946ms FAIL (2025-07-26 20:50:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 516.47677ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 225.823161ms FAIL (2025-07-26 20:50:52 UTC) orbiter warning: uplink: too many requests Duration: 225.869004ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 476.956503ms FAIL (2025-07-26 20:50:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 476.977418ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 648.001555ms FAIL (2025-07-26 20:50:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 648.044234ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 395.689907ms FAIL (2025-07-26 20:50:14 UTC) orbiter warning: uplink: too many requests Duration: 395.823285ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 710.079181ms FAIL (2025-07-26 20:50:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 710.133809ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 555.692454ms FAIL (2025-07-26 20:50:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 555.725479ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 1.123994517s FAIL (2025-07-26 20:49:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.124027751s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 913.117517ms FAIL (2025-07-26 20:49:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 913.17083ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 520.905669ms FAIL (2025-07-26 20:49:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 520.942309ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 229.53934ms FAIL (2025-07-26 20:49:18 UTC) orbiter warning: uplink: too many requests Duration: 229.575211ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 607.9541ms FAIL (2025-07-26 20:49:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 608.040541ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 1.337312851s FAIL (2025-07-26 20:48:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.33743436s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 225.19836ms FAIL (2025-07-26 20:48:52 UTC) orbiter warning: uplink: too many requests Duration: 225.289055ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 1.131050183s FAIL (2025-07-26 20:48:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.131105994s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 492.969641ms FAIL (2025-07-26 20:48:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 493.000692ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 495.387235ms FAIL (2025-07-26 20:48:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 495.417321ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 738.891874ms FAIL (2025-07-26 20:48:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 738.95064ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 436.988327ms FAIL (2025-07-26 20:47:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 437.018194ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 238.907908ms FAIL (2025-07-26 20:47:50 UTC) orbiter warning: uplink: too many requests Duration: 238.935828ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 734.224122ms FAIL (2025-07-26 20:47:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 734.272806ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 1.708181479s FAIL (2025-07-26 20:47:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.708211692s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 565.695335ms FAIL (2025-07-26 20:47:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 565.718413ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 1.35028375s FAIL (2025-07-26 20:46:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.350318037s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 541.186424ms FAIL (2025-07-26 20:46:39 UTC) orbiter warning: uplink: too many requests Duration: 541.21668ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 749.055307ms FAIL (2025-07-26 20:46:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 749.091041ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 501.92324ms FAIL (2025-07-26 20:46:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 502.152408ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 630.431381ms FAIL (2025-07-26 20:46:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 630.465056ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 476.308373ms FAIL (2025-07-26 20:45:59 UTC) orbiter warning: uplink: too many requests Duration: 476.338707ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 608.434343ms FAIL (2025-07-26 20:45:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 608.476827ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 452.384207ms FAIL (2025-07-26 20:45:56 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 452.414823ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 673.541706ms FAIL (2025-07-26 20:45:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 673.579517ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 273.672799ms FAIL (2025-07-26 20:44:56 UTC) orbiter warning: uplink: too many requests Duration: 273.697889ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 553.025352ms FAIL (2025-07-26 20:44:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 553.098432ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 536.197862ms FAIL (2025-07-26 20:44:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 536.254372ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 677.225952ms FAIL (2025-07-26 20:44:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 677.331104ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 761.548434ms FAIL (2025-07-26 20:44:17 UTC) orbiter warning: uplink: too many requests Duration: 761.618416ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 1.734856497s FAIL (2025-07-26 20:44:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.734947542s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 3.757608714s FAIL (2025-07-26 20:44:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 3.757639861s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)